If you are building evals for coding agents, at some point the question stops being "did it change the code" and becomes "did it write the tests". Coverage is the obvious metric and the obviously wrong one: a test that calls a function without asserting anything still counts as coverage. Mutation testing is the honest replacement — mutate the code, see whether the tests notice — and on a small module it is now cheap enough to run per diff, seconds to a minute.
So the metric becomes a mutation score: mutants killed over mutants tried. That is a ratio, and what it is a ratio of is the whole question.
The number I got before writing a test
I spent a morning reading a Python repo's mutation tooling: a script that generates mutants with cosmic-ray, runs each one against the tests that coverage says touch the changed lines, and reports what survived. On one 144-line module the authors had published the breakdown of a single full run:
| count | |
|---|---|
| mutants | 342 |
| killed | 237 |
| survived | 48 |
| survived (string-only) | 8 |
| crashed | 49 |
Run that through the score they were adding — killed over tried, with crashes excluded because no test has to notice a crash:
score = 237 / (237 + 48 + 8) = 0.809
Nobody had written a test yet. That 0.809 is the existing suite, measured on mutants drawn from that module.
What the ratio cannot say
Now suppose you use that ratio to grade an agent: give it a change with tests to write, run the mutation score on the diff, record the number. Two failures hide in the arithmetic.
The kills are not attributable. The runner executes every test that coverage credits with the changed lines — including the ones already in the file. So on a change inside a well-covered area, most of the kills belong to tests that predate the agent, and the score is high whatever the agent wrote. On a change in a thin spot, the same quality of new test scores low. Two diffs of the same size can differ by more than the effect you are trying to measure, and the difference is a property of the module, not of the agent.
The function that prints the ratio is usually the one that cannot attribute. Mutation runners are fastest when they stop at the first failing test: short circuit, record "killed", move on. That is exactly what you want for the human-facing question ("do these tests catch this break?") and exactly what you lose for the eval question ("which test caught it?"). In the tool I read, the fast path passes -x to pytest, so a mutant is recorded as killed with no record of the killer — while a second function in the same file, the one that builds the full kill matrix, runs without -x and stores the sorted list of failing tests per mutant. Both numbers exist. The ratio was computed from the one that throws the attribution away.
Record the delta, not the ratio
The eval number you want is the difference between two runs over the same mutant set:
- what the tests already there kill, and
- what those plus the agent's tests kill.
The difference — mutants no old test kills that a new test does — is immune to how well covered the module was before, and it is what "the agent wrote tests for this change" actually claims. A delta of zero then reads correctly as "the existing tests already killed these", where the raw ratio renders it as a 0.8 pass.
It costs one more run of the same matrix, not a new instrument. If the runner already stores killers per mutant, the subtraction is a set difference; if it only stores outcomes, one run over the union of old and new tests and one over the old tests alone gets you there.
The general shape
A ratio shipped without its reference set is a number with two causes. The reference is almost always one more run of the same instrument, and what it buys is the ability to say whose contribution the difference is.
This is not specific to mutation testing. "The agent fixed 7 of 10 issues" — against what baseline? "The prompt change cut latency by 40%" — from which distribution? The reading you can act on is the one that moves only when the thing you changed moves.
If you take one thing from the number at the top: before the score means anything, compute it on the code as it was. 0.809 was the starting line, and every agent on that module started from it.
Top comments (3)
Evaluating test-writing agents on raw mutation score conflates baseline test coverage with the agent's actual contribution. If a legacy test suite already kills 8% of generated mutants across the module, an agent can append vacuous assertions and still claim a high score.Attributing kills directly to the agent's new test functions through a delta matrix is the only honest way to isolate regression detection. A secondary trap is equivalent mutants. When an operator inverts an unused internal guard condition that cannot alter downstream program output, the mutant survives regardless of test quality. Filtering out unkillable equivalent mutants before computing the attribution delta keeps the benchmark from penalizing the agent for dead branch logic.
Thanks — that is the article's argument stated more compactly: the ratio is a property of the suite, so what an eval needs is the difference between two runs of the same mutant set, before and after the agent's test.
Two wrinkles from building it:
The delta needs a stable mutant identity, not a count. Each mutant has to be named (operator + position), or a diff that shifts the mutant set makes the delta meaningless.
Equivalent mutants are already inside the arithmetic, not only in the filtering step. In the harness I measured, string-literal survivors are listed apart as "most often equivalent" and still counted in the denominator — so a fully covered change touching three string literals loses exactly three points. Same shape as your inverted unused guard.
The per-test matrix is the expensive route: that harness kills per mutant over the covering set with -x, so no test is ever named. Attribution is free at mutant granularity.
cool!