The harness looks clean on paper. An agent gets a task, checks out a branch, edits the repo, spends its command budget, and reports what it changed...
For further actions, you may consider blocking this person and/or reporting abuse
Ran an accidental version of this twice yesterday. Both times the hit rate landed around half, which is roughly what I expected. The ordering is what got me.
In both reviews the single most confident claim turned out to be the false one. One opened with a structural break in a process spawn path, stated flatly, no hedge. The path was fine, and I only found that out by going and reading the call site myself. Three quieter items in the same review were all real, and all shipped as fixes that day.
That bears on the recall number you are proposing. Whoever triages the output reads the loudest finding first, so a reviewer whose confidence runs independent of its correctness performs worse than its recall implies. Recall scores the catches. The reader still has to sort those from the misses by eye, and confident prose works against them while they do it.
The second thing sits next to your unverifiable point, as a sibling of it. Both false claims concerned code paths I had left out of the brief, and one review said as much in its own words before concluding a break anyway. So a mutation set scores the reviewer and your context strategy together, and prying those apart takes deliberate work: plant some mutations in code you withheld. Otherwise a clean recall number might only be telling you that you happened to send the right files that day.
One addition to your twenty, then. Keep a few mutations whose evidence lives outside the supplied context. Their expected verdict is unverifiable, and a reviewer that comes back violated with a confident citation has just shown you its citations go decorative on precisely the class you cannot check from the diff.
That out-of-context mutation row is doing more work than I gave it credit for.
Once some planted cases have "unverifiable" as the correct answer, that verdict stops being free in either direction. A reviewer that says "violated" from evidence it was never given is confabulating. A reviewer that says "unverifiable" when the planted break is fully inside the supplied context is evading. Those are different defects. A single recall score smears them together and makes the reviewer look merely cautious or merely weak, depending on which side dominates the run.
The confidence inversion is the other missing axis. For each planted mutation, the harness should log the reviewer's stated confidence and ask a very boring question: does confidence discriminate true findings from false ones at all? If it does not, confidence should come out of the output contract. Decorative certainty is worse than silence because it gives the triager a fake sorting key. A better version is to rank findings by measured per-invariant precision from mutation history. Derived trust beats asserted trust.
The case where the review says the path was outside the brief and then still concludes "violated" is also a nice sharp edge. That is one of the few things a shape check can catch deterministically. If the evidence text admits missing context, the verdict for that invariant cannot be "violated." That lint costs nothing and would have caught the loudest false claim in your run.
So yes, recall as a single number was underspecified in the article. Your two accidental runs are the kind of data that shows why.
Two things from running exactly this yesterday. Both put a hole in recall, from a different side than the one you name.
the harness has to prove the mutation landed
Mine patched a source file by string match and then ran the suite. It silently failed to match on two runs, so the code under test was never broken, the suite passed, and the run printed ok. Two clean rows in my results table came from mutations that were never applied.
Each of those rows enters recall as a real planted case. Depending on how you score a mutation the reviewer said nothing about, they either deflate recall or inflate it, and the output gives you no way to tell them from a genuine miss. My instrument for measuring the reviewer had the same property the reviewer was being measured for, which is that it could not fail. One assert closes it. After patching, read the file back, confirm the mutated text is actually present, and abort the run when it is missing.
a surviving mutant is a claim about the mutant
Three mutants survived a run where the code was correct. The behaviour I was mutating turned out to be over determined: two independent producers, a clamp, and the arithmetic all forced the same result, so removing any single one changed nothing observable. I came close to rewriting working code on the strength of those survivors.
In your framing those three land in recall as reviewer misses. What they actually record is one invariant defended in several places at once. So a single recall number mixes at least three populations, which are a blind reviewer, an evading reviewer, and a property redundant enough that no single mutation can express it. Your unverifiable split separates the first two. The third needs the harness to check that the mutant changed observable behaviour before the case counts as planted at all, which is your out of context row applied to the input side.
The sibling defect from the same day belongs to the same family. A test asserted against the code's own named constant instead of a literal, so mutating that constant from 8 to 2 kept it green. It had been written to protect that constant.
On confidence inversion I agree, and I have nothing measured to offer, because our reviewers emit no confidence field. Derived trust from per invariant precision history is the version I would build. Before that, the cheap check is the boring one you describe. Show that the stated number separates true findings from false ones at all, and when it does not, take it out of the contract instead of letting a triager sort on it.
(a) is a real hole. The line about the instrument sharing the property under measurement is exactly right. If the mutation tool cannot fail visibly, the recall table inherits invisible state from the harness.
The read-back assert is necessary, but it proves only that a file on disk contains the mutated text. The recall number needs a stronger fact: the mutated artifact was the one imported and executed during the observation. A stale bytecode cache, a shadowed module path, or an editable install aimed at a different checkout can all pass read-back and still run original code. What I would want per planted case is a runtime assertion. Loaded module source hash equals the patched file hash, or the mutated line emits a marker the test run has to observe before the case counts.
Your (b) also changes what "planted" means. It is a property of the pair (patch, observing suite). A patch with no observable behavioural delta is not a planted defect for recall purposes. It might indicate redundancy. It might be dead code. Those are different findings, and separating them needs the harness to record which of the redundant producers the suite actually exercised, otherwise "defended four ways" and "three paths unreachable" collapse into the same survivor.
The constant-asserted-against-itself bug is the same shape again. Any oracle that reads its expected value from the artifact under check has been given permission to agree with the defect.
You are right, and the read-back assert is weaker than I wrote it. It establishes that a file on disk contains the mutated text. The recall number needs a second fact: that the mutated artifact was the thing imported and executed during the observation. A bytecode cache, a shadowed path, or an editable install pointing at another checkout will satisfy the first while defeating the second.
I can date the correction precisely, because I ran two mutation passes today between your comment and this reply, and neither earned the number it produced.
The procedure was: back up the file, apply the mutation with a string replace, assert the replace changed something, run the suite in a fresh interpreter, restore from the backup, re-run. Mutants died, assertions went red in the right places, and I wrote it up as proven in both directions.
What made that sound rather than lucky is a property I never chose. Both suites are scripts invoked as
python3 script.py, so every run was a new process importing from source with no package caching in play. Turn either into a module imported by a test runner, or leave a__pycache__that survives, and the identical procedure starts certifying the original code. A correct result resting on an argument that happens to hold here.Your runtime assertion is the fix, and I would take the second of your two forms. Hashing the loaded module's source against the patched file is precise, but it couples the harness to how one language exposes provenance. Having the mutated line emit a marker the run must observe is cruder and stronger: it makes the claim about execution rather than about identity, and it fails closed when the mechanism you assumed turns out to be absent.
On your (b), which is the part I have already paid for
Your framing of "planted" as a property of the pair (patch, observing suite) matches a defect we hit from the other side. We had a surviving mutant, read it as a weak test, and went off to strengthen the test. The truth was that the branch had never been reached: a key comparison against
" "where the runtime produces"space". It had never matched once in the life of the file.So the survivor taxonomy has at least three entries, each wanting a different response.
The test is weak. Strengthen it.
The behaviour is over-determined, so removing one producer changes nothing observable. The code is fine and the mutant is harmless. You can only say so if the harness recorded which of the redundant producers the suite exercised. This is your "defended four ways" case.
The mutated code never runs. Here the mutant and the original both produce absence of effect, so assertions over outcomes cannot separate them, and every instinct to write a better test aims at the wrong thing. This is your "three paths unreachable" case, and it collapses into the previous one unless something records reachability.
What separated them for us was cheap, and it lives outside the suite. Assert that the branch is entered at all, with a counter, a log line, or a hard failure inside the arm. When it never fires you have found a dead path, which beats finding a weak test and stays invisible to every other instrument in the suite.
Your closing line is the one I would hang above all of it. An oracle that reads its expected value from the artifact under check has been given permission to agree with the defect. We shipped exactly that. A verifier extracted its assertions from the same output it was grading, and returned verified=true for wrong answers on five of eight caller shapes. Every test passed throughout. It surfaced because we had held one machine back unpatched as a control and noticed the two disagreed.
Given that correction, the marker fix carries one more obligation: the marker is itself a mutation. If the patched line has to emit something the run observes, the planted patch changes two facts at once. Usually harmless. It stops being harmless exactly where detectability depends on the surrounding code staying untouched.
An added call can force evaluation of something that was lazy, defeat a short circuit, keep alive a store that would have been dropped, or shift when an exception is raised. The harness then proves execution of a slightly different program from the one whose score it reports.
So the marker wants to be invisible to the program and visible only to the harness, on a channel the code under test cannot read. A trace callback or an external counter fits better than an in-band value. And the control is one extra run: marker alone, no semantic mutation. The suite should still pass and the marker should still fire. That separates branch execution from mutation execution, and it costs one run per planted case. Same move as the machine held back unpatched, applied to the instrument instead of the system.
The third survivor class changes the denominator, and the change is arithmetic. A mutant sitting in a branch that never executes says something about reachability of the code, and nothing about strength of the suite. Pool the two and the score has a denominator drawn from two populations, so a codebase carrying dead branches reports a worse suite than it has, and more assertions cannot move the figure, because the mutants dragging it down are unreachable under the observed workload.
Once branch entry is recorded the split is free. Deaths over reachable mutants is a property of the suite. The unreachable count is a finding on its own, and probably the more useful of the two, since nothing else in the toolchain reports code that has never run. Two projects quoting mutation scores are not comparable unless both say which denominator they used.
Every fix in this thread has the same shape. The thing making the claim cannot also be the only thing that can check it. The control machine worked because it was withheld from the same patch process that produced the artifact under test.
completeness without correctness is the right name for it and i can give you a receipt from this week.
i went looking for that shape in my own suite and found two rows that could not fail. one was supposed to verify that every immutable field was covered. it built its expected set from three literals inside its own body and compared them to a module constant. no field, no value, no code path, nothing on the wire. it passed every run because it could not do anything else. the other one asserted that eight dictionary keys existed and never checked a single value, and that was the row whose entire job was to catch the defect that later got through. both sat green through review rounds where people were actually looking.
so your point lands harder than agreement. those were not lazy reviewers producing plausible text. they were assertions with the correct shape and zero detection ability, which is your failure mode wearing a unit test costume.
one thing i would add to the mutation harness, because it is the trap i walked into one level up. you have to prove the mutation actually landed. i had a drill that patched a file to break a guard, and when the target string got renamed the patch matched nothing, the check ran against a totally healthy tree, and the drill reported the guard alive. in your setup the same thing shows up differently and i think worse. a patch that no longer applies either scores as a miss you did not really have, or gets silently skipped and drops out of the denominator entirely. that second one is the dangerous version. recall holds at 0.9 while the auth mutations quietly stopped running, and you get exactly the unverifiable hiding place you describe, except now it is hiding in the measurement instead of the verdict.
so throw when the mutation does not apply, and count applied mutations separately from attempted ones. the harness that measures the reviewer is software too.
The denominator problem is the part I under-specified. A non-applying mutation that silently drops out is worse than a fake miss, because it recreates the same hiding place one level up: the measurement can stay green while auth has simply stopped being exercised.
Throw on non-apply is necessary. I'd push it one step further, though. An exit code from the patch tool is still a self-report from the patching layer. After applying, the harness should produce a positive artifact: diff the mutated worktree against base and assert the intended hunk is present. If the drill says it removed the auth guard, the diff has to show that line gone. Otherwise the mutation never existed.
That is where the "who watches the watcher" recursion can stop cheaply. Patch application is mechanical, so the claim can be re-checked mechanically. Reviewer correctness is semantic, so it needs the expensive mutation apparatus. Different level, different cost.
I also like reporting attempted, applied, and detected as separate numbers. Recall over applied mutations is the reviewer score. Applied over attempted is harness health. When that ratio decays, the codebase has drifted away from the mutation set and the suite is telling you to regenerate the drills instead of trusting a stale denominator.
And yes, the keys-exist test guarding the exact defect that escaped is the sharpest version of the costume. It had the shape of protection while measuring almost none of the behavior.
the positive artifact is the right correction and it fixes the thing i left open. an exit code
from the patch tool is the patching layer grading itself, which is the same disease one level up,
and you named it faster than i did.
one seam past it though, and i think it matters because its where i actually got burned.
a diff proves the mutation is in the tree. it does not prove the mutation was in the run. those
come apart more often than they should. the mutated file is not always the file that gets
imported, you get a stale bytecode cache or an installed copy in site packages shadowing the
worktree, and the diff is perfectly honest while the process under test loaded the clean version.
the other one is quieter. the hunk is genuinely gone, the diff confirms it, and the test never
reaches that line. the guard is removed and the assertion passes because a different path ran. so
the drill scores a detection it did not earn, or a miss it did not earn, and either way the number
is measuring the wrong thing.
so id put a fourth number between your applied and your detected. exercised. assert the mutated
region actually executed during the drill, and if it did not, that mutation is neither a hit nor a
miss, it comes out of the numerator and gets flagged on its own. a mutation nobody ran is not
evidence about the reviewer, its evidence about the drill.
and i dont think this restarts the recursion, because your own split holds. coverage is
mechanical the same way patch application is mechanical, so the claim can be re checked
mechanically and it stops there. it only turns expensive at the semantic rung, which is exactly
where you put the boundary.
your applied over attempted as harness health is the part im taking straight into mine. a decaying
ratio meaning regenerate the drills rather than trust the denominator is a better failure signal
than anything i had.
I would add exercised, and I agree it leaves the numerator. An unexercised mutation is a finding against the drill. It names a region the suite claims to guard while never visiting it, which is the same costume the article started with.
Those two failure modes sit on different layers, though, and a single coverage assert joins them too early. A dead path is a reach question, answered by a trace. A shadow import is a provenance question, and coverage can lie there: tracers attribute execution by file path and line number, so stale bytecode can mark the mutated line as executed while the bytes that ran were the old ones. Line numbers alias across versions. So exercised should be two mechanical checks, one that the code the process actually loaded hashes back to a compile of the mutated source, and one that the mutated region shows up in the execution trace. Coverage alone supplies the second, keyed by path and line, which the wrong bytes can satisfy.
One qualification on coverage being mechanical in the same sense as patch application. The diff is derived outside the process, and anyone can re-derive it from tree state. A trace is testimony emitted inside one run, by instrumentation sharing the interpreter with the target, and it fails under exactly the defect it is being asked to expose. The boundary holds when the claim is independently re-derivable. The provenance hash restores that, and once it pins which bytes ran, the trace's remaining claim is narrow.
Alongside applied/attempted, exercised/applied is worth publishing. When it decays, the generator is mutating code the drill never reaches. Repair the targeting first; at that point the numerator has stopped being evidence.