This is the second walkthrough in a series.
The first built a gate: a Stop hook that will not let Claude Code end its turn while any test is failing, so it cannot call a job done on a red suite.
This one asks whether the tests behind that green are worth passing, and you do not need to have read the first to follow along.
What you will do: measure how many real bugs your test suite can actually catch. In the worked example, a green suite catches just one planted bug in ten; among the nine it misses is a discount that quietly becomes a surcharge. A tighter case shows the sharper trap: a test can cover every line of a function and still notice nothing. You will watch both scores land, then hand Claude Code the holes and make it close them. About twenty-five minutes for the worked example; a first pass on your own repository takes your whole suite’s runtime once for every mutant, so start with your smallest tested file.
Who this is for: you gate Claude Code, or any coding agent, on green tests, and the suite has been reassuringly green ever since.
Who should skip it: if you already run mutation testing, this is your Tuesday. Step 3, turning your existing survivor backlog into an agent work order, may not be.
Never written a test? Your version of the whole method is at the end: ten minutes, three planted errors, no code.
You need: Python 3 (3.9 or later, standard library only, nothing to install). Claude Code for step 3; steps 1 and 2 run without it.
Contents
Watch a test do nothing
Count what your suite can see
Hand the survivors to the agent
Then:
When it still goes wrong
Proving it on code you ship
If you never write code.
I broke a production module of mine on purpose, 105 small ways, one at a time, and ran its test suite after every break. The suite is real: 21 regression tests, all green, guarding an 822-line file my own automation depends on. The tests noticed 38 of the 105 breaks. The other 67, one of them on a line the suite executes every single run, would have shipped without a sound.
That experiment is the whole method, and it answers a question a passing test suite cannot. The gate proves the tests pass. It cannot prove the tests are worth passing; a test that asserts nothing sails straight through it. And if your instinct is to have a second agent check the tests, then a third to check the second, the regress ends here instead, at an experiment rather than another opinion: break the code on purpose, and see whether the alarm rings.
1. Watch a test do nothing
(You know why an assert-nothing test passes? Skip to step 2; this section is the on-ramp.)
Here is the toy calculator shop from last time, one walkthrough later. Its add works, and the suite is two small tests. The gate is green. (If you download the companion folder, calc.py also carries a new pricing function, which is step 2’s problem, and check.sh, the last walkthrough’s gate script, along for the ride. Leave both alone for now.)
calc.py
def add(a, b):
return a + b
test_calc.py
import unittest
from calc import add
class TestAdd(unittest.TestCase):
def test_basic(self):
self.assertEqual(add(2, 3), 5)
def test_zero(self):
self.assertEqual(add(0, 0), 0)
# ...the unittest.main() entrypoint is unchanged below
Save both files in one folder and open your terminal there; the import from calc import add only resolves if they sit together. (Downloading the folder in step 2 does this for you.) One of these two tests is doing almost all of the work, and one of them is doing almost none. You can find out which in thirty seconds. Break add on purpose (return a - b, the classic bug) and run the suite, python3 -m unittest, the same command the gate runs: test_basic fails. Now, with the code still broken, run only the other test:
$ python3 -m unittest test_calc.TestAdd.test_zero
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
Green, on code that subtracts. add(0, 0) is 0 whether add adds, subtracts, or multiplies, so test_zero approves all three. It has one way to fail, and almost no wrong version of the code triggers it. Note what your coverage tool would say about it: test_zero executes every line of add, one hundred per cent, top marks. Coverage measures whether tests run the code. It has no opinion on whether they would notice anything. test_zero has sat in that suite looking exactly as load-bearing as test_basic, and it would wave the classic bug straight through the gate on its own.
A test you have never watched fail is not yet a test.
Test-driven veterans have said a version of that for twenty years: never trust a test you haven’t seen fail. The discipline is the same, and the move is the one you just made: break the code on purpose, and the test either notices or it does not. Once, by hand, takes thirty seconds. Every plausible break is a script. Fix add back before you move on; the script is next.
2. Count what your suite can see
The shop grew this week. Claude Code added a pricing function, and the suite is green, which is all the gate checks. Here is the function:
def discounted_total(price, quantity, discount_percent):
"""Total cost of an order, in pounds.
Orders of 10 or more items get discount_percent knocked off.
"""
total = price * quantity
if quantity >= 10:
total = total * (1 - discount_percent / 100)
return round(total, 2)
Money code. A boundary, a formula, a rounding rule: three places to be quietly wrong. (Yes, real tills count integer pence rather than floating-point pounds; hold that thought, it returns at the end.) The question you cannot answer by reading the green gate: if one of those went wrong tonight, would any test notice?
mutate.py answers it by brute honesty. It is a transparent teaching instrument, 180 lines of standard library you can read top to bottom, built to make the mechanism visible on one file rather than to replace a mature framework. Its whole engine is the dozen classic ways code goes wrong that it knows how to plant, one character at a time:
OP_SWAPS = {
ast.Add: ast.Sub, # a + b -> a - b
ast.Mult: ast.Div, # a * b -> a / b
ast.GtE: ast.Gt, # >= -> > (the off-by-one at every boundary)
ast.Eq: ast.NotEq,
ast.And: ast.Or,
# ...the reverse of each, plus < and <=, a dozen swaps in all,
# and integer nudges: 10 -> 11
}
For each place in your file where one of those swaps applies, it makes that one change (a mutant of your code), runs your whole suite, restores the file, and records the verdict. Before any of that it runs your suite once on the untouched code, timed: a red baseline gets refused outright, because against a suite that is already failing, every verdict is noise. Then the two outcomes. A mutant that makes at least one test fail is killed : the alarm rang. A mutant that leaves every test green survived : that exact bug could ship tonight.
Grab the folder , hosted on my own site; it is short, dependency-free Python you can read before you run it. Unzip it and open your terminal inside the tests-worth-passing folder; every command below runs from there, no setup.
The four working files are calc.py, test_calc.py, mutate.py, and check.sh (last walkthrough’s gate, along for the ride); a README and the agent’s finished tests sit alongside them and need nothing from you. When you are ready to point this at your own code, the folder’s AUDIT.md carries the reusable version: the audit protocol, a survivor-triage worksheet, and the command for your language.
Now stop before you run it, and put a number down. Your suite is green and it passes. Of ten deliberate breaks to this file, how many do you think it catches? Hold that guess against the result:
$ python3 mutate.py calc.py
10 mutants of calc.py · suite: python3 -m unittest -q
baseline: suite green in 0.2s (per-mutant timeout 60s)
1 KILLED line 2: a + b -> a - b
2 SURVIVED line 10: price * quantity -> price / quantity
3 SURVIVED line 11: quantity >= 10 -> quantity > 10
4 SURVIVED line 11: 10 -> 11
5 SURVIVED line 12: total * (1 - discount_percent / 100) -> total / (1 - discount_percent / 100)
6 SURVIVED line 12: 1 - discount_percent / 100 -> 1 + discount_percent / 100
7 SURVIVED line 12: 1 -> 2
8 SURVIVED line 12: discount_percent / 100 -> discount_percent * 100
9 SURVIVED line 12: 100 -> 101
10 SURVIVED line 13: 2 -> 3
Score: 1/10 killed, 9 survived.
Every SURVIVED line is a change to your code that your whole test suite
cannot tell from the version you meant to write.
The sixth mutant is the one to read twice. 1 - discount_percent / 100 became 1 + discount_percent / 100: a customer’s 20 per cent discount becomes a 20 per cent surcharge , and the gate stays green. The third is the boundary: >= became >, the customer buying exactly ten items loses the discount you promised them, green. Nine ways for money code to be wrong, and the suite from step 1 sees none of them, because nothing in it ever calls the new function. The gate never lied. “The tests pass” was true every time it said so. It just was not the thing you needed to be true.
A fair objection: nothing tests that function, so a plain coverage report would have flagged it too, without any of this mutant theatre. True, and if that were all mutation testing found, you would not need it. So run the case coverage cannot see. Delete test_basic, keep only test_zero, and run the instrument again:
$ python3 mutate.py calc.py
10 mutants of calc.py · suite: python3 -m unittest -q
baseline: suite green in 0.1s (per-mutant timeout 60s)
1 SURVIVED line 2: a + b -> a - b
...
Score: 0/10 killed, 10 survived.
...
The subtraction bug now survives, on a function with one hundred per cent line coverage. Your coverage dashboard reports add fully tested; the instrument reports that no test would notice if it subtracted. Coverage tells you the code ran. A kill tells you a lie got caught. They are different instruments, and only one of them is measuring what you care about.
The same function can be one hundred per cent covered and zero per cent detected.
This is not a toy-only failure, and it is the shape to hold on to. On the 822-line module I opened with, the survivor that stung most was exactly this: a covered line, inside a function the suite runs every single time, that no assertion actually pins. Put test_basic back and carry on.
One reading note before you run it on anything you love: killed is the good outcome. Every killed mutant is a bug class your suite would catch, so on this one screen a KILLED is the line you are hoping for, even though the word sounds like something broke.
This move is called mutation testing, and it is older than most of the code you have ever shipped. Breaking one character at a time is a fair stand-in for the elaborate bugs real code grows, because of the coupling effect : catch the small, dumb faults and you catch the large subtle ones as a by-product. 1 The nine survivors here are the ones your suite missed, and every one of them is now a job you can hand to the thing that wrote the tests.
3. Hand the survivors to the agent
Nine survivors is a work order, addressed to the thing that wrote the tests. Paste the nine SURVIVED lines into Claude Code, followed by this instruction:
These mutants survived mutation testing: the suite stays green when any one of these changes is made to calc.py. Write tests in test_calc.py that kill them. Do not modify calc.py or mutate.py. Then run
python3 mutate.py calc.pyand keep going until the score is clean.
When I ran exactly that, the agent came back with four tests and a habit I did not ask for: it annotated each one with the wrong answer the mutant would produce. The survivor list had turned into a specification it could compute against.
def test_discount_applies_at_exactly_ten_items(self):
# Boundary: quantity == 10 qualifies. 10.0 * 10 = 100, minus 20% = 80.0.
# Kills the > / >=, threshold 10->11, and every discount-formula mutant:
# total / (1 - d/100) -> 125.0
# total * (1 + d/100) -> 120.0
self.assertEqual(discounted_total(10.0, 10, 20), 80.0)
def test_rounds_to_two_decimal_places(self):
# 3.333 must round to 3.33; a round(total, 3) mutant returns 3.333.
self.assertEqual(discounted_total(3.333, 1, 0), 3.33)
Two of its four tests are above; the finished suite, those four plus the two you started with, ships as calc_solution_tests.py in the folder (named without the test_ prefix so it stays out of your way until you want it), so you can run it yourself instead of retyping it from the excerpts. To see the clean score without doing step 3, drop that file in as test_calc.py and run python3 mutate.py calc.py. Left as it downloads, the folder still scores 1/10, because the suite the gate runs holds only the two starting tests.
Then I ran the instrument again myself, because you never take the agent’s word for a score when you can take the score’s word for it:
$ python3 mutate.py calc.py
...
Score: 10/10 killed, 0 survived.
Exit code 0, the instrument’s all-clear. Same code, same tests passing as before, and now green means something it did not mean before: ten classic ways to break this file, and a test rings for every one. (My run went clean in one round, but this toy is small. When yours does not, paste what survived straight back and go again; and for the rare survivor no round can kill, the closing section shows the other move: you let it live, with a comment.)
One step remains, and it is the one that actually ends the regress: read the four tests it wrote, and check every asserted value against what the code should do, not against what it does. The distinction is load-bearing. An agent kills mutants by pinning current behaviour; look back at its annotations and you can see the 80.0 was computed from the implementation. On correct code that is exactly what you want. On code that is already wrong, the same move canonises the bug as specification, with a clean mutation score as its alibi.
The score proves the tests ring when the code changes; only your read proves they are ringing for the truth.
What the instrument buys you is that the read is bounded: four short tests with a known purpose, instead of an unbounded hope about a whole suite.
Two honest notes before you rely on it.
Where it earns its keep. A fresh agent asked cold to “add tests” for that function, with no survivor list, scored 10/10 on its first try. On a four-line function whose docstring names the threshold, the odds are friendly, and the survivor loop buys you little. That is not the point. The point is that on real code you cannot tell whether it guessed well by looking, and the survivor list is what turns “write better tests” from a vague instruction into a measurable work order. The payoff climbs exactly where you cannot eyeball it: a forty-line function, a docstring that has drifted from the code, a module you inherited and half-trust.
None of this is new, and that is the reassuring part. Handing a test-writer a list of surviving mutants is called mutation-guided test generation , and search-based tools were doing it a decade before anyone had an LLM; the agent is just a better test-writer than they were. Google found the part that matters for you here: engineers act on mutants delivered as review-time tickets and quietly ignore the same mutants dumped in a batch report. 2 The survivor list is that first channel, a work order a developer acts on. Meta has published a version of this pattern with an LLM at the test-writing end. 3 You are running the individual-developer version of a documented industrial practice.
When it still goes wrong
It is slow on a real repo. Every mutant is a full suite run, so keep it out of the Stop hook: the gate fires every turn, this audit runs when the code or tests change. Industrial tools go further, mutating just the lines a change touches, which is how Google runs it at scale.
A clean score is a bounded claim. 10/10 covers only this tool’s small vocabulary of breaks. Real gaps live outside it: money in binary floats that no swap can expose, and “kills” that are really crashes, not caught assertions. The score narrows the worry; it does not end it.
Some survivors cannot be killed. An equivalent mutant reads differently but behaves identically, so no test can catch it. If you cannot name an input that would expose a survivor, it may be one, and it is allowed to live with a comment.
The score is a worklist. Optimise the percentage like a KPI and you get tests engineered to twitch at mutants rather than to state what the code should do, which is the disease “the tests pass” had, one level up. Chase named survivors; ignore the number.
The verdict contradicts the file you are reading. You are running a stale compiled copy of the file you mutated. Delete
__pycache__ortouchthat file. (mutate.py already gives its own runs a fresh cache.)Not on Python? The instrument is language-specific; the method is not. Reach for mutmut on Python, PIT on the JVM, Stryker on JavaScript and TypeScript, cargo-mutants on Rust. The rule holds everywhere: break the code, count the catches.
Prove it on code you ship
Pick the smallest file in your own project that has tests you trust. Four moves before you run it:
Confirm a green baseline. The instrument refuses a red baseline, because against a failing suite every verdict is noise.
Point
TEST_COMMANDat your test runner. It sits at the top of mutate.py, defaulting to plain unittest. On pytest, swap that one line, keeping it a list of arguments, not a string:[sys.executable, "-m", "pytest", "-q", "tests/test_foo.py"].Narrow it to the file you are mutating. Aim at just the tests that exercise that file, not the whole suite. Every mutant runs the command once, so a wide one is the difference between a coffee and an afternoon, and the reason people quit after one slow run.
Predict, then compare. Write down the score you expect before you look. The gap between the number you predicted and the number you got is the most useful thing this walkthrough produces.
Here is the anatomy of the number I opened with, re-run the morning this published so it is evidence and not a memory. The module: 822 lines of my own automation, 21 green regression tests. The run: 105 mutants, 38 killed, 67 survived. Where the 67 hid is the whole lesson. Point coverage at the same suite and the module reads 43 per cent covered; 49 of the survivors sit in code no test executes at all, which a coverage report already flags for you. The other 18 are the ones only mutation can see, because they sit on lines coverage counts as covered.
One function holds most of them: a weekly counter the suite genuinely imports, calls, and runs green, whose single test asserts that the count is a non-negative integer and checks nothing else. So flip the sign on its seven-day window and it looks a week into the future, the count silently collapses toward zero, and all 21 tests still pass. That is test_zero from step 1 wearing a production badge: a fully-covered function that no assertion pins , in my own code, caught by the instrument you just ran on a toy.
So I did step 3 on my own code. I handed that survivor to the agent with the same work order, and it wrote the test the counter never had: stage a single item inside the window, then assert the count is exactly one, not merely non-negative. I applied the sign-flip by hand and ran both assertions.
current_weekly_promotion_count() = 0 (one item, aged 1 day, window = 7 days)
assert count >= 0 -> PASS (the assertion that was already there)
assert count == 1 -> FAIL (the assertion the agent just wrote)
The old check stayed green on a function that now counts nothing, exactly as it had all along. The new one went red, because a window flipped a week into the future finds zero where it should find one. One survivor, handed over and killed, on the first test that pinned a number instead of trusting a sign.
The fix for most of the others is that same move: pin the number the code should return , so a window that flips or a boundary that slips finally has somewhere to fail.
One survivor on that function is the exception. The window admits an item on mtime >= cutoff, and to tell >= from > you need a file whose modification time lands on the exact cutoff instant. The cutoff is read from the clock, so the only way to hit that instant is to mock the clock, and I judged that test double heavier than the bug it would catch. So it stays alive, with a comment naming why.
It is worth being precise about what it is, because it is not an equivalent mutant: an equivalent mutant is one no input can expose, and this one has an input, a mocked clock, that I have simply declined to write.
That is step 3 on the days the loop does not go clean: some survivors earn a real assertion , some earn a refactor ticket , one earns a comment explaining why it lives, and telling which is which is the work. If your score stings, the sting is information, the first honest measurement your suite has ever had.
Then add this rule to your CLAUDE.md, the standing instructions your agent reads each session, so the discipline survives you forgetting it:
When you fix a bug: first write a test that fails on the broken code,
show me it failing, then fix the code. A test I have never seen fail
does not count as coverage.
That is the manual mutant from step 1, promoted to standing policy: every bugfix now arrives with proof that its test can ring.
You are holding three instruments now, and most arguments about testing are really an argument between two of them. Coverage asks whether a line ran. Mutation asks whether a break in that line would be caught. Your own read asks whether the behaviour a passing test pins is the one you actually wanted.
Coverage measures reach. Mutation measures detection. Only your read decides what is right.
The gate keeps the agent honest about whether the tests pass; mutation and your own read are what keep you honest about whether they are worth passing. Green stops being where the question ends. It moves from are the tests passing to what broken versions did these tests catch.
If you never write code
Break the code, count the catches: the method never depended on Python. It works on anything an agent hands back where being wrong is checkable, and the case you most need it for is not code at all. It is the claim.
Here is the whole thing with no code in it. Take a report or a summary you know cold. Copy it, and plant a few deliberate errors in the copy, the kind of wrong that would cost you something if it slipped through. Hand the copy to your “review this” agent, and count what it catches.
I ran it on one of my own weekly stats summaries before publishing. In a copy I changed a signup count from four to six, moved a date back by a day, and added a confident sentence that one post had our weakest open rate, which it did not. Then: “check this against the raw export.” It caught all three, and flagged a fourth claim I had written carelessly and never planted.
Three planted, three caught , and the review earned my trust the same way step 1’s test did: I had watched it catch mistakes I controlled before I believed the ones I did not. A review you have never watched catch a planted error is worth exactly what a test you have never watched fail is worth.
The catches are your kills; the misses are the claims you must never hand over unchecked. Code is the easy case, because a machine runs it and a mutant is unambiguous. The claims your agent writes when there is no suite to run, the summary and the analysis and the report, are the harder case, and the one worth an instrument of its own: one that has to find the errors you did not think to plant, checking each claim against its source instead of against a copy you already know is wrong. That is where this goes next.
Related: How Reliable Is Your AI Agent makes the broader case, The Stable Liar shows the failure mode up close, and Never Let Claude Code Tell You It’s Done builds the test gate this piece audits.
What was your score, and which survivor surprised you? The answers steer what gets built next.
Richard DeMillo, Richard Lipton and Frederick Sayward, "Hints on Test Data Selection: Help for the Practicing Programmer," IEEE Computer 11(4), 1978, 34–41. The paper that introduced mutation analysis and, with it, the coupling effect; the idea itself is usually traced to a 1971 class paper of Lipton's.
Goran Petrović and Marko Ivanković, "State of Mutation Testing at Google," Proceedings of ICSE-SEIP 2018. Two findings carry into this piece: mutation analysis is made tractable across Google's roughly two billion lines by mutating only the lines a change touches; and engineers act on mutants surfaced as review-time diffs while ignoring the same mutants in a batch report.
Christopher Foster et al., "Mutation-Guided LLM-based Test Generation at Meta," arXiv:2501.12862, 2025. Meta's ACH system plants undetected faults, then has an LLM write the tests that kill them.
Top comments (0)