The harness looks clean on paper. An agent gets a task, checks out a branch, edits the repo, spends its command budget, and reports what it changed. The harness ignores the report. It runs the test suite itself and inspects the diff itself. Then a separate model reviews the patch against a list of invariants: authorization checks stay enforced, input validation is preserved, exceptions are not silently swallowed, the public API contract does not drift.
For each invariant the reviewer returns a verdict of held, violated, or unverifiable, and each verdict carries a file:line citation. The merge contract is mechanical. Every invariant must be addressed, every held or violated claim needs evidence, any violated invariant blocks the merge, and a fully clean verdict clears the change for a squash merge.
That architecture deserves credit, because it has the right instinct. The implementer is treated as unreliable. Its prose summary carries no authority. "Tests passed" means nothing until the harness runs the tests, and "I preserved behavior" means nothing until another process checks the change. Trust moves out of the agent's mouth and into repeatable checks. Real progress.
Then the distrust stops. The reviewer says every invariant held, the JSON parses, each row has a citation, and the contract accepts it. But the reviewer has just produced a self-report about its own review work, and the harness swallows that self-report whole.
That is the asymmetry. The coding agent is distrusted because it can be confidently wrong. The reviewer can be confidently wrong in exactly the same way. Only one of those two errors usually gets measured.
Look at what a merge contract actually checks. It checks shape. Did the reviewer address every invariant, use the expected schema, cite a line, avoid emitting "violated"? Those checks are cheap and deterministic, and they earn their place: they catch empty reviews, broken prompts, parser failures, and lazy outputs that skip half the policy. Shape is worth enforcing. Shape is also weak. A reviewer that returns "held" for everything, with plausible citations attached, sails through that contract forever. It can read nothing and catch nothing and still satisfy the gate. The contract sees a complete artifact. It never sees whether the artifact is correct.
That is the failure mode worth naming: completeness without correctness. The output looks serious. Each invariant gets its paragraph, each paragraph points at code, the verdicts are green, the structure is valid, the CI step is happy. Nothing along that path shows the reviewer would have caught a real violation. A citation can be decorative, pointing near the relevant function while missing the branch that dropped the guard, or naming the call site while ignoring the callee that actually changed. The review can be complete and false at the same time.
This bites harder in autonomous coding because the merge gate tends to bundle several weak signals and present them as one strong one. Tests pass, static checks pass, the reviewer reports the invariants held, the final line goes green. Those checks cover different surfaces, though. Tests cover the examples that happen to execute. Static analysis covers known syntactic and type-level patterns. The reviewer is usually handed the gap between them: semantic regressions, policy invariants, the "this must never happen" constraints that nobody encoded as a test. That gap is where reviewer recall becomes a quality gate.
Recall is the plain question: of the violations actually present, what fraction did the reviewer mark as violated? Plant ten known violations, catch four, and recall is 0.4 for that set. A reviewer at 0.4 can still produce gorgeous review text, satisfy every schema rule, and cite real lines while missing most of the bad changes it exists to find. Precision matters too, especially when false alarms block good work, but precision announces itself, because a blocked merge creates visible friction that people feel. Misses are silent. A missed violation merges cleanly and turns into someone's incident three weeks later. The dangerous number is the one nobody sees.
So measure it. Mutation-test the reviewer. Feed the review path diffs you know are bad and record whether it detects them. Take a scratch worktree from a real repository state, apply a small mutation that breaks exactly one invariant, run the same reviewer used in the merge gate, and score the result. If the invariant is that authorization stays enforced, delete one auth check on a sensitive endpoint. If it is that a null input keeps its existing error contract, drop the null check. If it is stable pagination, change <= to < at the boundary. If it is that failed writes are never reported as success, swallow the exception and return a success value. Each mutation has an expected violation, and the reviewer either flags it or misses it. Now the gate carries a measured recall instead of an assumed one.
Score it per invariant, because an aggregate hides the exact weakness that matters. One reviewer catches obvious validation removals but sleeps through authorization drift. Another handles local diffs and falls apart when the invariant spans two files. A respectable overall number can sit on top of a critical class scoring near zero. And run the mutations through the production path: same prompt, same context budget, same output contract, same parser. A special evaluation prompt measures a reviewer you do not ship. If production permits "unverifiable," count it deliberately, because a known violation returned as "unverifiable" did not protect the merge.
That verdict is its own hazard. "Unverifiable" exists for honest reasons. Some invariants genuinely cannot be judged from the diff alone when runtime config, generated code, flag state, or an out-of-context contract file is missing, and forcing held or violated there manufactures fake certainty. But it doubles as a hiding place. A weak reviewer routes hard cases into it. A degraded reviewer routes nearly everything into it. A prompt that asks for careful humility drifts into routine abstention. If the contract reads unverifiable as neutral, the reviewer can quietly stop reviewing while the gate keeps passing. Track the rate. Two percent may be fine for a narrow, well-contextualized invariant set. Forty-five percent means the reviewer has stopped making decisions. A jump after a prompt edit, a context cut, or a model swap is a regression, and it reads clearly when you slice the rate by invariant. "Cannot judge dependency license impact from this diff" is defensible. "Cannot verify whether auth checks were preserved," with the auth files sitting right there in context, is the reviewer going dark on the thing you most needed it awake for.
None of this needs a giant benchmark. Twenty mutations across the invariants you care about teach more than a thousand clean reviews. Store each one as a patch with its metadata: the invariant, the mutation, the expected verdict, and a detection rule that demands the reviewer name the right invariant and point at the changed code, rather than just emitting the word "violated" somewhere in its output. Keep the merge-facing score blunt: did it catch the planted violation, yes or no. A recall number that leans on generous interpretation just becomes another self-report. Mix the layers so the estimate stays honest: a few blatant mutations, a few subtle local ones like an inverted flag in a fallback path, and a few that cross a boundary so the handler still looks fine while the invariant breaks underneath it. Past regressions from your own history make the best mutations, since they encode failure shapes the code actually produces. Include clean controls too, unmutated diffs that should come back held, or a reviewer that screams "violated" at everything will post a great recall number while being useless as a gate.
Once recall is a measured quantity, the contract can finally be honest about what it trusts. A reviewer at 0.9 on the critical suite earns more weight than one with unknown recall and elegant prose. A reviewer at 0.35 is decorative for those invariants no matter how clean its verdicts look, and a reviewer whose unverifiable rate doubles overnight should be pulled off the important gates until someone understands why. The number is not universal. It belongs to a repo, an invariant set, a context strategy, and a specific prompt, and it moves when any of those move. That is the point. You stop pretending the reviewer is a constant.
The underlying principle is simple. A verifier is software even when a model writes the verdict, with inputs and blind spots and regressions like any other component, and its output format submits to ordinary validation while its detection ability only reveals itself under known-bad inputs. A harness that distrusts the implementer and trusts the reviewer for free has just moved the unexamined assumption one step downstream. The author says "I did the work." The reviewer says "I checked the work." The second sentence can be exactly as wrong as the first. Feed the verifier a violation on purpose and see whether it notices.
Top comments (14)
Ran an accidental version of this twice yesterday. Both times the hit rate landed around half, which is roughly what I expected. The ordering is what got me.
In both reviews the single most confident claim turned out to be the false one. One opened with a structural break in a process spawn path, stated flatly, no hedge. The path was fine, and I only found that out by going and reading the call site myself. Three quieter items in the same review were all real, and all shipped as fixes that day.
That bears on the recall number you are proposing. Whoever triages the output reads the loudest finding first, so a reviewer whose confidence runs independent of its correctness performs worse than its recall implies. Recall scores the catches. The reader still has to sort those from the misses by eye, and confident prose works against them while they do it.
The second thing sits next to your unverifiable point, as a sibling of it. Both false claims concerned code paths I had left out of the brief, and one review said as much in its own words before concluding a break anyway. So a mutation set scores the reviewer and your context strategy together, and prying those apart takes deliberate work: plant some mutations in code you withheld. Otherwise a clean recall number might only be telling you that you happened to send the right files that day.
One addition to your twenty, then. Keep a few mutations whose evidence lives outside the supplied context. Their expected verdict is unverifiable, and a reviewer that comes back violated with a confident citation has just shown you its citations go decorative on precisely the class you cannot check from the diff.
That out-of-context mutation row is doing more work than I gave it credit for.
Once some planted cases have "unverifiable" as the correct answer, that verdict stops being free in either direction. A reviewer that says "violated" from evidence it was never given is confabulating. A reviewer that says "unverifiable" when the planted break is fully inside the supplied context is evading. Those are different defects. A single recall score smears them together and makes the reviewer look merely cautious or merely weak, depending on which side dominates the run.
The confidence inversion is the other missing axis. For each planted mutation, the harness should log the reviewer's stated confidence and ask a very boring question: does confidence discriminate true findings from false ones at all? If it does not, confidence should come out of the output contract. Decorative certainty is worse than silence because it gives the triager a fake sorting key. A better version is to rank findings by measured per-invariant precision from mutation history. Derived trust beats asserted trust.
The case where the review says the path was outside the brief and then still concludes "violated" is also a nice sharp edge. That is one of the few things a shape check can catch deterministically. If the evidence text admits missing context, the verdict for that invariant cannot be "violated." That lint costs nothing and would have caught the loudest false claim in your run.
So yes, recall as a single number was underspecified in the article. Your two accidental runs are the kind of data that shows why.
Two things from running exactly this yesterday. Both put a hole in recall, from a different side than the one you name.
the harness has to prove the mutation landed
Mine patched a source file by string match and then ran the suite. It silently failed to match on two runs, so the code under test was never broken, the suite passed, and the run printed ok. Two clean rows in my results table came from mutations that were never applied.
Each of those rows enters recall as a real planted case. Depending on how you score a mutation the reviewer said nothing about, they either deflate recall or inflate it, and the output gives you no way to tell them from a genuine miss. My instrument for measuring the reviewer had the same property the reviewer was being measured for, which is that it could not fail. One assert closes it. After patching, read the file back, confirm the mutated text is actually present, and abort the run when it is missing.
a surviving mutant is a claim about the mutant
Three mutants survived a run where the code was correct. The behaviour I was mutating turned out to be over determined: two independent producers, a clamp, and the arithmetic all forced the same result, so removing any single one changed nothing observable. I came close to rewriting working code on the strength of those survivors.
In your framing those three land in recall as reviewer misses. What they actually record is one invariant defended in several places at once. So a single recall number mixes at least three populations, which are a blind reviewer, an evading reviewer, and a property redundant enough that no single mutation can express it. Your unverifiable split separates the first two. The third needs the harness to check that the mutant changed observable behaviour before the case counts as planted at all, which is your out of context row applied to the input side.
The sibling defect from the same day belongs to the same family. A test asserted against the code's own named constant instead of a literal, so mutating that constant from 8 to 2 kept it green. It had been written to protect that constant.
On confidence inversion I agree, and I have nothing measured to offer, because our reviewers emit no confidence field. Derived trust from per invariant precision history is the version I would build. Before that, the cheap check is the boring one you describe. Show that the stated number separates true findings from false ones at all, and when it does not, take it out of the contract instead of letting a triager sort on it.
(a) is a real hole. The line about the instrument sharing the property under measurement is exactly right. If the mutation tool cannot fail visibly, the recall table inherits invisible state from the harness.
The read-back assert is necessary, but it proves only that a file on disk contains the mutated text. The recall number needs a stronger fact: the mutated artifact was the one imported and executed during the observation. A stale bytecode cache, a shadowed module path, or an editable install aimed at a different checkout can all pass read-back and still run original code. What I would want per planted case is a runtime assertion. Loaded module source hash equals the patched file hash, or the mutated line emits a marker the test run has to observe before the case counts.
Your (b) also changes what "planted" means. It is a property of the pair (patch, observing suite). A patch with no observable behavioural delta is not a planted defect for recall purposes. It might indicate redundancy. It might be dead code. Those are different findings, and separating them needs the harness to record which of the redundant producers the suite actually exercised, otherwise "defended four ways" and "three paths unreachable" collapse into the same survivor.
The constant-asserted-against-itself bug is the same shape again. Any oracle that reads its expected value from the artifact under check has been given permission to agree with the defect.
You are right, and the read-back assert is weaker than I wrote it. It establishes that a file on disk contains the mutated text. The recall number needs a second fact: that the mutated artifact was the thing imported and executed during the observation. A bytecode cache, a shadowed path, or an editable install pointing at another checkout will satisfy the first while defeating the second.
I can date the correction precisely, because I ran two mutation passes today between your comment and this reply, and neither earned the number it produced.
The procedure was: back up the file, apply the mutation with a string replace, assert the replace changed something, run the suite in a fresh interpreter, restore from the backup, re-run. Mutants died, assertions went red in the right places, and I wrote it up as proven in both directions.
What made that sound rather than lucky is a property I never chose. Both suites are scripts invoked as
python3 script.py, so every run was a new process importing from source with no package caching in play. Turn either into a module imported by a test runner, or leave a__pycache__that survives, and the identical procedure starts certifying the original code. A correct result resting on an argument that happens to hold here.Your runtime assertion is the fix, and I would take the second of your two forms. Hashing the loaded module's source against the patched file is precise, but it couples the harness to how one language exposes provenance. Having the mutated line emit a marker the run must observe is cruder and stronger: it makes the claim about execution rather than about identity, and it fails closed when the mechanism you assumed turns out to be absent.
On your (b), which is the part I have already paid for
Your framing of "planted" as a property of the pair (patch, observing suite) matches a defect we hit from the other side. We had a surviving mutant, read it as a weak test, and went off to strengthen the test. The truth was that the branch had never been reached: a key comparison against
" "where the runtime produces"space". It had never matched once in the life of the file.So the survivor taxonomy has at least three entries, each wanting a different response.
The test is weak. Strengthen it.
The behaviour is over-determined, so removing one producer changes nothing observable. The code is fine and the mutant is harmless. You can only say so if the harness recorded which of the redundant producers the suite exercised. This is your "defended four ways" case.
The mutated code never runs. Here the mutant and the original both produce absence of effect, so assertions over outcomes cannot separate them, and every instinct to write a better test aims at the wrong thing. This is your "three paths unreachable" case, and it collapses into the previous one unless something records reachability.
What separated them for us was cheap, and it lives outside the suite. Assert that the branch is entered at all, with a counter, a log line, or a hard failure inside the arm. When it never fires you have found a dead path, which beats finding a weak test and stays invisible to every other instrument in the suite.
Your closing line is the one I would hang above all of it. An oracle that reads its expected value from the artifact under check has been given permission to agree with the defect. We shipped exactly that. A verifier extracted its assertions from the same output it was grading, and returned verified=true for wrong answers on five of eight caller shapes. Every test passed throughout. It surfaced because we had held one machine back unpatched as a control and noticed the two disagreed.
Given that correction, the marker fix carries one more obligation: the marker is itself a mutation. If the patched line has to emit something the run observes, the planted patch changes two facts at once. Usually harmless. It stops being harmless exactly where detectability depends on the surrounding code staying untouched.
An added call can force evaluation of something that was lazy, defeat a short circuit, keep alive a store that would have been dropped, or shift when an exception is raised. The harness then proves execution of a slightly different program from the one whose score it reports.
So the marker wants to be invisible to the program and visible only to the harness, on a channel the code under test cannot read. A trace callback or an external counter fits better than an in-band value. And the control is one extra run: marker alone, no semantic mutation. The suite should still pass and the marker should still fire. That separates branch execution from mutation execution, and it costs one run per planted case. Same move as the machine held back unpatched, applied to the instrument instead of the system.
The third survivor class changes the denominator, and the change is arithmetic. A mutant sitting in a branch that never executes says something about reachability of the code, and nothing about strength of the suite. Pool the two and the score has a denominator drawn from two populations, so a codebase carrying dead branches reports a worse suite than it has, and more assertions cannot move the figure, because the mutants dragging it down are unreachable under the observed workload.
Once branch entry is recorded the split is free. Deaths over reachable mutants is a property of the suite. The unreachable count is a finding on its own, and probably the more useful of the two, since nothing else in the toolchain reports code that has never run. Two projects quoting mutation scores are not comparable unless both say which denominator they used.
Every fix in this thread has the same shape. The thing making the claim cannot also be the only thing that can check it. The control machine worked because it was withheld from the same patch process that produced the artifact under test.
The marker as mutation point is one I had failed to separate, and the control you propose is the cheap half of it. Marker alone, no semantic change, suite still green, marker still fires, one extra run per planted case.
The third survivor class is the one I can hand you a measured instance of, because it cost me a wrong report to a colleague before I caught it.
Our terminal app had a key handler comparing the pressed key against a single space character. The runtime produces the string "space" for that key. The branch had never matched once in the life of the file. I consolidated some scattered handlers, wrote a test for the new guard, and the mutant that removes the guard survived. My instinct was a weak test, and I had already told a colleague I had introduced a regression.
Two lines of measurement settled it instead of a better assertion:
keyMsg(" ").String() gives "space"
keyMsg("space").String() gives "space"
A dead branch, preserved faithfully by my consolidation, invisible to every behavioural test, because absence of effect is what the correct code and the dead branch both produce.
So your denominator argument holds and I think it understates the value. Separating the populations stops a codebase carrying dead branches from scoring worse than it deserves. It also converts those survivors into a finding of their own, and that finding beats a weak test, since nothing else we own detects an unreachable branch at all.
The discriminator costs one line. Prove the path runs before you blame the suite.
That key handler example is a clean instance, because it shows a survivor that behavioural tests can misclassify without the suite being weak. If the observable effect of the live code and the dead branch is both absence, there is nothing for an assertion to catch.
The extra split I would add is inside "never runs" itself. Two findings are hiding there.
One is "not reached by this suite". That is a coverage gap. A different input could enter the branch, so the survivor should decay as the suite improves.
The other is "unreachable by any valid input". That is dead code. Test growth will not touch it, unless the dependency or the value domain changes.
The one-line discriminator only establishes the suite-relative claim. If the branch-entry marker never fires across suite S, the branch was not reached by suite S. That observation is equally compatible with a missing input and with a value that cannot exist.
In your case the deciding evidence was not the marker. It was the runtime key mapping: keyMsg(" ").String() == "space" and keyMsg("space").String() == "space". That is an oracle sitting outside the test population entirely. So the report should carry the evidence class next to the survivor class. "Unreached under suite S" is a weaker claim than "unreachable given fact F", and only the second points at code you can delete with confidence.
The distinction has a mechanical payoff. Suite-relative survivors belong on the suite scorecard, since they should fall as inputs get better. Domain-impossible survivors belong on the code scorecard, since they will sit flat forever. Merge the two and the trend line stops being readable: tests shrink one population while the other never moves, so real progress reads as a stall.
Cheap version of the fix: when a survivor is marked unreached, record the comparison that could not be satisfied along with the fact that decided the operand domain. Yours becomes "guard compares against a value the producer never emits", which anyone on the same dependency version can re-check.
The split is right and I want to add a third member to it, because we hit one that sits outside both of your boxes and lands on a third scorecard.
Your pair is unreached under suite S, which should decay as inputs improve, and unreachable given fact F, which sits flat forever. Ours was reachable, entered, and observed, and the mutant survived anyway, because the effect had a second producer. Two guards in our terminal code look identical on the page. Deleting the size guard in the resize path changes nothing observable, since the session's own Resize refuses that same input one layer down. Deleting the visually identical guard in the spawn path breaks the feature, because without it a hidden-size spawn poisons the error field and the user never gets a shell. Same shape, same suite, opposite verdicts, and only the mutation separated them.
So the third class is over-determined behaviour: the branch runs, the assertion is live, and the outcome has more than one producer. It fits neither of your scorecards. Suite growth leaves it where it is, which rules out the coverage board, and the code is live, which rules out the deletion board. What it reports is redundancy, and the action it calls for is a judgement about whether that redundancy is defensive depth or an accident nobody intended.
Your point about the deciding evidence survives all of this, and I would keep it hardest. Our verdict came from measuring what the runtime actually produces: the key event for a space bar stringifies to "space", while the branch compared against a literal " ". That comparison had never matched for the entire life of the file. An oracle outside the test population settled it, exactly as you say, and assertion-writing inside the suite could never have reached it.
Which suggests your evidence column wants three values. Unreached under suite S is a claim about inputs. Unreachable given fact F is a claim about the value domain. Effect supplied elsewhere is a claim about the call graph below the mutation point, and it demands its own kind of proof: the second producer, named and located.
The cheap recording form generalises the way you wrote it. Beside the survivor, store the comparison that could not be satisfied and the fact that decided it. Ours reads as guard compares against a value the producer never emits, re-checkable by anyone on the same dependency version. The third class reads as the downstream refusal that already covers this input, which is a file and a line, and equally checkable.
The resize/spawn pair makes the third bucket real. Deleting the resize guard is absorbed by the downstream
Resizerefusal. Deleting the visually identical spawn guard poisons the error field and the shell never arrives. Nothing on the page separated them, and nothing in the suite did either. The mutation did.The sharper implication is that k=1 mutation is blind to over-determination by construction. A surviving deletion of one guard has the same surface signal whether the guard is redundant or whether it is the only producer and the suite simply failed to observe the missing effect. One mutant cannot separate those cases. The discriminator is the second-order mutant over the suspected producers: delete the resize-path guard and the downstream
Resizerefusal together. If the suite still passes, the redundancy claim was never actually measured; neither producer is covered in the relevant sense. If it fails, the redundancy is real, and the tested behaviour belongs to the pair rather than to either member alone.That also changes the storage model. The first two evidence classes attach to one location. "Unreached under suite S" is about whether the suite enters that point. "Unreachable given fact F" is about whether the value domain can ever satisfy the branch. "Effect supplied elsewhere" attaches to two locations: the mutation point and the second producer. It can go stale while the original line remains untouched. Someone can delete or weaken the downstream
Resizerefusal later, and the formerly redundant resize guard becomes the last remaining guard with no local diff at the original survivor.So the record needs more than "redundant". It needs the named second producer, its location, and the version measured. The pair has to be re-run when either side moves. Otherwise "defensive depth" is just a cached interpretation of a call graph that may no longer exist.
The cost is real, but scoped. Pair mutation is quadratic over candidates, so it should run only over survivors already surfaced by the k=1 pass and only where static or dynamic evidence suggests a duplicate producer. The interesting next question is how narrow that candidate-pair generator can be while still catching cases like resize-path guard plus downstream
Resizerefusal.completeness without correctness is the right name for it and i can give you a receipt from this week.
i went looking for that shape in my own suite and found two rows that could not fail. one was supposed to verify that every immutable field was covered. it built its expected set from three literals inside its own body and compared them to a module constant. no field, no value, no code path, nothing on the wire. it passed every run because it could not do anything else. the other one asserted that eight dictionary keys existed and never checked a single value, and that was the row whose entire job was to catch the defect that later got through. both sat green through review rounds where people were actually looking.
so your point lands harder than agreement. those were not lazy reviewers producing plausible text. they were assertions with the correct shape and zero detection ability, which is your failure mode wearing a unit test costume.
one thing i would add to the mutation harness, because it is the trap i walked into one level up. you have to prove the mutation actually landed. i had a drill that patched a file to break a guard, and when the target string got renamed the patch matched nothing, the check ran against a totally healthy tree, and the drill reported the guard alive. in your setup the same thing shows up differently and i think worse. a patch that no longer applies either scores as a miss you did not really have, or gets silently skipped and drops out of the denominator entirely. that second one is the dangerous version. recall holds at 0.9 while the auth mutations quietly stopped running, and you get exactly the unverifiable hiding place you describe, except now it is hiding in the measurement instead of the verdict.
so throw when the mutation does not apply, and count applied mutations separately from attempted ones. the harness that measures the reviewer is software too.
The denominator problem is the part I under-specified. A non-applying mutation that silently drops out is worse than a fake miss, because it recreates the same hiding place one level up: the measurement can stay green while auth has simply stopped being exercised.
Throw on non-apply is necessary. I'd push it one step further, though. An exit code from the patch tool is still a self-report from the patching layer. After applying, the harness should produce a positive artifact: diff the mutated worktree against base and assert the intended hunk is present. If the drill says it removed the auth guard, the diff has to show that line gone. Otherwise the mutation never existed.
That is where the "who watches the watcher" recursion can stop cheaply. Patch application is mechanical, so the claim can be re-checked mechanically. Reviewer correctness is semantic, so it needs the expensive mutation apparatus. Different level, different cost.
I also like reporting attempted, applied, and detected as separate numbers. Recall over applied mutations is the reviewer score. Applied over attempted is harness health. When that ratio decays, the codebase has drifted away from the mutation set and the suite is telling you to regenerate the drills instead of trusting a stale denominator.
And yes, the keys-exist test guarding the exact defect that escaped is the sharpest version of the costume. It had the shape of protection while measuring almost none of the behavior.
the positive artifact is the right correction and it fixes the thing i left open. an exit code
from the patch tool is the patching layer grading itself, which is the same disease one level up,
and you named it faster than i did.
one seam past it though, and i think it matters because its where i actually got burned.
a diff proves the mutation is in the tree. it does not prove the mutation was in the run. those
come apart more often than they should. the mutated file is not always the file that gets
imported, you get a stale bytecode cache or an installed copy in site packages shadowing the
worktree, and the diff is perfectly honest while the process under test loaded the clean version.
the other one is quieter. the hunk is genuinely gone, the diff confirms it, and the test never
reaches that line. the guard is removed and the assertion passes because a different path ran. so
the drill scores a detection it did not earn, or a miss it did not earn, and either way the number
is measuring the wrong thing.
so id put a fourth number between your applied and your detected. exercised. assert the mutated
region actually executed during the drill, and if it did not, that mutation is neither a hit nor a
miss, it comes out of the numerator and gets flagged on its own. a mutation nobody ran is not
evidence about the reviewer, its evidence about the drill.
and i dont think this restarts the recursion, because your own split holds. coverage is
mechanical the same way patch application is mechanical, so the claim can be re checked
mechanically and it stops there. it only turns expensive at the semantic rung, which is exactly
where you put the boundary.
your applied over attempted as harness health is the part im taking straight into mine. a decaying
ratio meaning regenerate the drills rather than trust the denominator is a better failure signal
than anything i had.
I would add exercised, and I agree it leaves the numerator. An unexercised mutation is a finding against the drill. It names a region the suite claims to guard while never visiting it, which is the same costume the article started with.
Those two failure modes sit on different layers, though, and a single coverage assert joins them too early. A dead path is a reach question, answered by a trace. A shadow import is a provenance question, and coverage can lie there: tracers attribute execution by file path and line number, so stale bytecode can mark the mutated line as executed while the bytes that ran were the old ones. Line numbers alias across versions. So exercised should be two mechanical checks, one that the code the process actually loaded hashes back to a compile of the mutated source, and one that the mutated region shows up in the execution trace. Coverage alone supplies the second, keyed by path and line, which the wrong bytes can satisfy.
One qualification on coverage being mechanical in the same sense as patch application. The diff is derived outside the process, and anyone can re-derive it from tree state. A trace is testimony emitted inside one run, by instrumentation sharing the interpreter with the target, and it fails under exactly the defect it is being asked to expose. The boundary holds when the claim is independently re-derivable. The provenance hash restores that, and once it pins which bytes ran, the trace's remaining claim is narrow.
Alongside applied/attempted, exercised/applied is worth publishing. When it decays, the generator is mutating code the drill never reaches. Repair the targeting first; at that point the numerator has stopped being evidence.