A green test suite proves the tests pass. It does not prove they would go red if the code were wrong.
mutant closes that gap the only way it can be closed: it breaks your code on
purpose, one token at a time, and checks whether your suite notices.
A mutation your tests don't notice is a line whose behaviour nothing asserts. Coverage would have called that line covered.
pip install -e . # from a clonePython 3.10+. No dependencies.
mutant run --tests "pytest -q" --src "src/**/*.py"Two arguments carry the whole idea:
--tests— the command that must fail when the code is wrong.--src— the files that get broken.
This repository ships a deliberately weak example: a small pricing module and a test suite that is green, reads as reasonable, and binds almost nothing. Every test in it asserts something true — they just all sit in the middle of the input range. The output below is copied verbatim from that run:
$ mutant run --tests "python -m pytest tests -q" --src "src/**/*.py" --root examples/weakly_tested --quiet
mutant · 1 file(s) · 19 mutation(s) planned
baseline: python -m pytest tests -q
baseline passed in 0.6s · per-mutant timeout 10s
Mutations your tests did not notice:
src/pricing.py:4 10 → 11 SURVIVED BULK_QUANTITY = 10
src/pricing.py:14 >= → > SURVIVED if quantity >= BULK_QUANTITY:
src/pricing.py:15 * → / SURVIVED total = total * (1 - BULK_DISCOUNT)
src/pricing.py:15 1 → 2 SURVIVED total = total * (1 - BULK_DISCOUNT)
src/pricing.py:15 - → + SURVIVED total = total * (1 - BULK_DISCOUNT)
src/pricing.py:16 2 → 3 SURVIVED return round(total, 2)
src/pricing.py:23 >= → > SURVIVED if subtotal >= FREE_SHIPPING_THRESHOLD:
src/pricing.py:37 <= → < SURVIVED if quantity <= 0:
src/pricing.py:37 0 → 1 SURVIVED if quantity <= 0:
src/pricing.py:38 continue → break SURVIVED continue
19 run · 9 killed · 10 survived · 0 error (8.9s)
binding score: 47.4 % (killed / decided)
Lines your test suite does not bind:
src/pricing.py → 4, 14, 15, 16, 23, 37, 38
A survivor is not automatically a bug — it is a line whose behaviour no test asserts. Decide, per line, whether that is intended.Read the survivors as sentences:
The entire bulk-discount rule can be deleted, inverted or turned into a division and every test still passes. Nobody has ever asserted what a bulk order costs.
Line coverage for that file is 100 %.
Python, JavaScript / TypeScript, Go, Rust, Java, C / C++, C#, Swift, Kotlin,
Scala, PHP, Dart, Ruby, shell. mutant is a lexical tool, not a parser:
it masks out strings and comments, then rewrites one operator. That is what
makes it work anywhere your test command works — and what limits it (see
Honest limits).
mutant run --tests "npm test" --src "src/**/*.ts"
mutant run --tests "go test ./..." --src "**/*.go"| Flag | Meaning |
|---|---|
--tests CMD |
the command that must fail for a mutation to count as caught |
--src GLOB |
files to mutate; repeatable (default src/**/*, lib/**/*) |
--exclude GLOB |
paths to skip; repeatable, adds to the built-in list |
--root DIR |
project root; the test command runs here |
--max-mutants N |
cap the run — a seeded sample spread across all files, not the first N |
--budget SECONDS |
stop after roughly this long and report what was decided |
--seed N |
reproduce an earlier sample exactly |
--timeout SECONDS |
per-mutant timeout (default: 3× baseline, min 10 s) |
--json PATH |
machine-readable report for CI |
--dry-run |
list what would be tried, run nothing |
--quiet |
summary only |
mutant operators lists all 24 operators. mutant restore recovers a tree
left mutated by an interrupted run.
| Code | Meaning |
|---|---|
0 |
every mutation was caught |
1 |
at least one mutation survived |
2 |
misuse — bad paths, or a tree that needs manual attention |
3 |
the baseline failed — a red suite cannot judge anything |
- run: mutant run --tests "pytest -q" --src "src/**/*.py" --budget 300 --json mutant.jsonBudget it. A mutation run costs one full test run per mutation; that is the
price of the answer, and --budget is how you decide how much of it to pay.
Three decisions are worth knowing about, because each of them is a way this tool could have quietly lied to you.
A red baseline is refused, not tolerated. If your suite is already failing,
every mutant "fails the tests" too, and every mutation looks caught. The score
would be a perfect 100 % and completely meaningless. mutant exits 3.
Errors are excluded from the score. A mutation that could not be evaluated is not evidence in either direction. Folding it into the denominator would let a broken run look like a well-bound one.
Your tree is restored, and it is restored by the next run. No signal
handler can help you here: SIGKILL, a closed terminal and a power cut never
run one. So before the first edit, the original bytes go into
.mutant-journal/, and recovery happens at the next startup. If a protected
file changed in a way mutant cannot attribute to itself — a colleague's edit,
a rebase — it is reported as foreign, left untouched, and the backup is
kept. Restoring over somebody else's work would be worse than the mess.
Running mutant on its own source is the only honest way to publish it:
mutant run --tests "python -m pytest -q" --src "src/mutant/*.py" --max-mutants 60 --seed 1| binding score | unbound files | |
|---|---|---|
| first measurement | 45.0 % | 8 |
| after closing the recovery gaps | 58.3 % | 7 |
The first run pointed straight at the code that decides whether your files come
back and whether somebody else's edit gets overwritten — every branch of the
recovery report was unbound. Those are now tested
(tests/test_recovery_reporting.py) and
cli.py has dropped off the list entirely.
The remaining 25 survivors are mostly cosmetic constants: how many findings to
list before "... and N more", the default column width, round(x, 3). They are
left alone deliberately. Writing tests to pin a truncation limit would raise the
number and improve nothing — which is exactly the trap the "the score is not a
target" warning below is about. The point of the list is to be read, not
emptied.
The measured example above originally reported 8 killed · 11 survived. It was
wrong. __pycache__ was serving bytecode compiled from a previous mutant:
many mutations preserve file length (+ for -), a mutant is written and
restored inside the same clock second, and CPython validates a .pyc against
exactly that (mtime, size) pair.
Consequences, both measured: verdicts were decided by stale bytecode during the run, and the example's source was left byte-identical to the original while importing it produced the mutant's answer.
The fix is PYTHONDONTWRITEBYTECODE=1 for the child plus a strictly increasing
mtime on every write. The regression test in
tests/test_bytecode_cache.py was verified to
fail without it — with the fix removed, all three mutations of a fully
pinned return 20 + 1 falsely survive, reporting a binding score of 0 % where
the truth is 100 %.
If your language has a build cache keyed on (mtime, size), the same trap is
waiting for any tool that rewrites files in place.
- It is a scanner, not a parser. Exotic input (regex literals holding quotes, nested template expressions) can make it skip a mutation or emit one that does not compile. Both are visible in the report; neither turns into false confidence.
- It runs sequentially. One test run per mutation, no parallelism. Use
--budgetand--max-mutantsrather than waiting. - A survivor is not automatically a bug. It is an unasserted line. Sometimes that is exactly what you meant.
- The score is not a target. Chasing 100 % produces tests written to kill mutants rather than to describe behaviour. Read the survivor list; ignore the number if it does not help you.
mutmut and cosmic-ray for Python, Stryker for JS/C#/Scala, and PIT for Java are deeper, AST-aware and language-specific. If you live in one language and want the thorough answer, use those.
mutant is for the other case: a quick, dependency-free, budget-bounded triage
that works in any repo with a test command, including polyglot ones — answering
"does my suite bind this at all?" in a couple of minutes rather than
"what is my exact mutation score?" overnight.
MIT — see LICENSE.