testpick

Run only the tests your change can actually break. Language agnostic, self-hosted, and it runs the whole suite whenever it cannot prove a test is safe to skip.

$ testpick select --since origin/main --format human

testpick  run 13 of 89 tests

  - 13 of 89 known tests touch the changed files

  1 changed file(s)
    src/refute/render.py 13 test(s)

  76 test(s) skipped

That is a real run against a real project. Breaking render.py on purpose and running only those 13 tests still caught the regression — which is the only promise that matters here.

---

The problem

Test impact analysis is a solved idea with an unsolved distribution. Every open source implementation is welded to one ecosystem:

Tool	Works with
pytest-testmon, pytest-impact	Python only
Tia	JUnit / Spock only
Various Maven plugins	Java only
jest --onlyChanged	Jest only

And the language-agnostic version is a proprietary SaaS: Datadog's Test Impact Analysis has no self-hosted option at all.

So the repositories that need this most — polyglot monorepos, where CI is slowest and a Python service, a Go worker and a TypeScript frontend all sit in one tree — are exactly the ones with nothing to use.

testpick is language agnostic because it never parses your language. It reads coverage data your test runner already produces, in formats that already exist, and does set arithmetic against a git diff.

How it works

per-test coverage  ──build──▶  impact map  ──select──▶  the tests to run
(you already have it)         (test → files)    ▲
                                                │
                                          git diff

Two commands:

# once per pipeline, or nightly on main
testpick build --from coverage:.coverage --map .testpick/impact.json

# on every pull request
testpick select --since origin/main --format pytest

Failing safe is the whole design

A test selector that skips a test which would have failed turns a red build green and nobody finds out until production. That is a far worse outcome than a slow pipeline, so testpick is default-deny. It runs the entire suite whenever:

• there is no map, or the map is unreadable, empty, or a schema it does not understand
• the map was built at a commit that is not an ancestor of HEAD — it describes a history that no longer exists
• the map is older than max_age_days (14 by default)
• a changed file matches always_run — lockfiles, CI config, conftest.py, Dockerfiles, build files
• a changed file is not in the map and not explicitly ignorable — a new file, an untested file, or a hole in the coverage data. testpick cannot tell which, so it does not guess.

The only risk it takes on your behalf is the ignore list — documentation, licences, images — and that list is short, boring, and printed by testpick explain.

Every decision comes with its reasoning attached:

$ testpick select --since origin/main
testpick: running everything - src/payments/refunds.py is not in the impact map
and not matched by any ignore pattern, so its blast radius is unknown

Install

pip install testpick     # or: pipx install testpick / uv tool install testpick

Python 3.9+, zero dependencies. testpick runs in CI where every added dependency is another thing that can break your pipeline at 3am.

Note the tool needs Python, but the tests it selects can be in any language.

Producing per-test coverage

testpick reads three formats. You need coverage recorded per test, not aggregated.

Python — coverage.py dynamic contexts:

pytest --cov=src --cov-context=test
testpick build --from coverage:.coverage

Anything emitting LCOV — JS, Go, Rust, C++. LCOV carries a TN: test-name record, so one file can hold many tests:

testpick build --from lcov:coverage/

Anything else — the escape hatch. If your runner can report what each test touched, it can feed testpick:

{ "suite::checkout flow": ["src/cart.ts", "src/pricing.ts"] }

testpick build --from json:testpick-input.json

Sources merge, which is how a monorepo works:

testpick build \
  --from coverage:services/api/.coverage \
  --from lcov:apps/web/coverage/ \
  --from json:services/worker/testpick.json

In CI

- name: Choose the tests
  id: pick
  run: |
    if testpick select --since origin/${{ github.base_ref }} \
         --format pytest > selected.txt; then
      echo "mode=subset" >> $GITHUB_OUTPUT
    else
      echo "mode=all" >> $GITHUB_OUTPUT   # exit 10: testpick could not narrow it down
    fi

- name: Run them
  run: |
    if [ "${{ steps.pick.outputs.mode }}" = "all" ]; then
      pytest
    else
      xargs -a selected.txt --no-run-if-empty pytest
    fi

Exit codes are the contract:

Code	Meaning
0	a selection was made — run exactly what was printed (possibly nothing)
10	testpick could not narrow it down — run the whole suite
3	testpick could not run

Keep a full run on your default branch. testpick is for shortening the pull request loop, not for replacing your merge gate. Selection is only ever as good as the map it was built from.

Inspecting the map

$ testpick stats
  tests         89
  source files  6
  tests/file    min 13, max 54, mean 33.3

  most-depended-on files (changing these runs the most tests)
       54  src/refute/source.py
       47  src/refute/extract.py

$ testpick explain src/refute/render.py
file  src/refute/render.py
touched by 13 test(s):
  ...
A change to this file selects exactly these tests.

stats doubles as an architecture smell test: a file that pulls in 90% of your suite is telling you something about your coupling.

Configuration

Drop a .testpick.json in your repo root — see examples/.testpick.json for a commented starting point. Patterns use fnmatch semantics against the repo-relative posix path, where * crosses directory separators, so *.md matches docs/guide/intro.md.

When a path matches both lists, always_run wins. That is deliberate: an ambiguous rule should send you toward running more tests, never fewer.

Limits, stated plainly

• The map goes stale. It records which files each test touched at the commit it was built at. Rebuild it regularly — nightly on your default branch is the usual answer — and keep max_age_days honest.
• File granularity, not line granularity. More precise in theory, far more fragile in practice: a moved line invalidates line-level data, while file level survives ordinary refactoring.
• Coverage misses non-import coupling. Tests that shell out to a binary, hit a network service, or depend on a file read at runtime may touch code that coverage never attributes to them. Put those paths in always_run.
• Not a merge gate. See above.

Contributing

The bug report we want most: testpick selected a subset and a skipped test would have failed. That is a correctness bug and outranks every feature. Include the map (testpick stats), the diff, and the testpick select --format json output. See CONTRIBUTING.md.

git clone https://github.com/rubenmaas/testpick && cd testpick
PYTHONPATH=src python -m unittest discover -s tests -t . -v

No dev dependencies. CI additionally dogfoods testpick on its own suite.

Funding

testpick is free and MIT licensed, and will stay that way.

It converts CI minutes back into engineering time, and on a large suite that is a measurable line on a bill. Sponsoring a fair share of what it saves you keeps it maintained: Sponsor.

Licence

MIT. See LICENSE.