Originally published on sergei-parfenov.com.
A bug fix can make every new test pass and still introduce a regression. Here is a deliberately const...
For further actions, you may consider blocking this person and/or reporting abuse
The mutation-testing note is the one part I would put a boundary on, because I ran it against your fixture. mutmut 3.7.0 on CPython 3.14.6, your corrected implementation as the only source file and just the first two checks as the suite: 5 mutants, 5 killed, 0 survivors. Same run against the plausible patch, the one with
if not statusesthat you already showed is wrong: also 5 mutants, 5 killed, 0 survivors. The operators it generated wereis Nonetois not None,list(orders)tolist(None), two string mutations on thestatuskey, andintonot in— the mutation you name as the useful one,statuses is Nonetonot statuses, is not in that set, since the operators rewrite a node in place rather than swap an identity test for a truthiness test. So the mutation score gives a perfect grade to both implementations under the suite that cannot separate them, which leaves your review question doing all the work rather than the tool.This is a really useful catch. You’re right, mutmut can give a perfect score here while missing the semantic mutation that actually matters. That makes the “plausible wrong implementation” check more important than the mutation score itself. Would you treat mutation tools as a supplement, with semantic mutants derived from the contract?
The distinction between a test that verifies the patch’s intent and one that verifies the contract is the key failure mode. I’d add one small review artifact for agent loops: keep the requirement, the assertion, and the observed output as separate records.
Before an agent writes or repairs a test, state the invariants and at least one neighboring case that could be confused with the reported bug. Then review the expected value against the requirement—not the current implementation—and run one deliberate negative or mutation case. If the test never goes red against a plausible wrong implementation, it has not demonstrated that it distinguishes the contract.
The same boundary shows up in browser agents: “the click returned successfully” is only an interaction result. The workflow should separately record the intended target, the page/context actually used, the action evidence, and an authoritative read-back of the postcondition. If navigation, reconnect or account drift makes that evidence stale, the result should be
unknown, not a green success or an automatic retry.Disclosure: I maintain Hronaut, a source-available local visible-browser workspace with local MCP and human takeover for sign-in, 2FA, CAPTCHA and consequential writes. I’m sharing it as one example of this evidence boundary, not as a claim that it solves the broader testing problem.
I like the idea of keeping the requirement, assertion, and observed output separate. It makes the evidence boundary much clearer and stops the current implementation from quietly becoming the oracle. Have you found a lightweight way to enforce this in agent loops without adding too much review overhead?
Your closing question also has a countable answer, and the count decides whether the next assertion is worth reading.
I ran your fixture against eight wrong implementations, one per contract axis instead of one per token:
return ordersinstead oflist(orders);not statuses; filter ignored entirely;order["status"] == statuses; reversed output; one pass per requested status (so["paid", "paid"]returns duplicates); a re-sort of the caller's list; and append-a-sentinel (every returned value is correct, the caller's list silently grows). Rejected:f(ORDERS) is not ORDERSTwo things fall out. The value of an added assertion is exactly the number of enumerated wrong implementations that only it rejects - my duplicate-status check (
f(ORDERS, ["paid", "paid"]) == [ORDERS[0]]) rejected nothing the order check did not already reject, so on this fixture it is decoration. And the surviving axes are the ones operators do not generate:list(orders)toreturn ordersis a one-click diff that passes your two checks, your third, and the fails-before/passes-after test; only an identity check, or a caller that mutates what it got back, separates them. Your fix writeslist(orders)deliberately and nothing in the fixture observes that decision.That is also the reading I would take from vinhnguyenthanhdn's mutmut run above: 5 mutants, 5 killed, identical for your corrected implementation and for the
if not statusesversion you already showed is wrong. Token-level operators on a five-line function mostly produce mutants that raise (list(None)) or move a string literal; they do not produce "right symptom, wrong contract", which is the failure mode this post is about. A 100% mutation score here is not evidence about the suite, it is evidence that the mutants were drawn from the code rather than from the requirements.For the agent loop I would upgrade step 1 accordingly: write the expected behavior down as three to five separable wrong implementations per requirement, with the axis named - empty vs. missing, identity, order, side effects on inputs - and treat the suite as acceptable only if it rejects each of them. The list is cheap to keep next to the issue, and it tells you which assertion to add: the one that rejects an entry no current check rejects.
This is a much stronger framing than my original step 1. I like the idea of treating each requirement as a small set of named wrong implementations, then adding assertions only when they kill a survivor. That also explains why 100% mutation coverage can still say almost nothing about semantic coverage here.
Would you keep those wrong implementations as explicit fixtures in the repo, or generate them from the contract during the agent loop?
The None versus empty list case is a good fixture precisely because the defect is not in the code. It is in a requirement that was never written down anywhere the patch author could see it. Two falsy values carrying different meanings tends to live in someone's head until an incident drags it out.
Your review question, which plausible wrong implementation would this test reject, is close to a working definition of what a test is for, and it is the part I will carry into teaching. I mark a lot of student test suites, and the most common failure has never been too few assertions. It is assertions derived from the code that was just written rather than from the behaviour that was asked for. All of them pass. None of them can fail. Agents did not invent that habit, they just made it fast enough to fill a repository with it before anyone reads a line.
Of your four steps I would weight the third heaviest. Once the repairer can edit both the implementation and the check, the loop is no longer converging on the requirement, it is converging on internal consistency, and internal consistency is cheap. Protecting the test command and the configuration alongside the file is the detail most setups miss, because skipping execution looks identical to passing in the log.
One question on the ExecCritic numbers. Do you read the drop from the weaker test agent as bad tests actively misdirecting repair, or as the loop spending part of its budget on test generation instead of repair attempts? With compute unmatched those are hard to separate, and they point at different fixes.
I don’t think the numbers let us separate those two effects. The loop clearly got worse with weaker test feedback, but with unmatched compute we can’t attribute the full drop to active misdirection. I’d read it as evidence that generated-test feedback can hurt in this setup, not proof of the mechanism. A matched-budget ablation would be the interesting next experiment.
I run a fleet of autonomous agents and this matches what I've seen concretely. The failure isn't that the tests are wrong - it's that a test which verifies the patch's intent (as the LLM wrote it) is indistinguishable from one that verifies the patch's contract. Your example is the classic case:
if not statusesreturns orders when the spec says omit-the-filter should too, so the generated test happily locks in the bug because it asserts the flawed behavior.What I'd add: in agent loops I've started checking the test itself against the spec, not against the code. Ask the agent to state the two invariants before writing the assertion, then diff the assertion against those invariants. Also, meta-reviewing a minimal failing test (RED first) catches a lot of these - if the agent never writes a RED test it's usually pure confirmation.
Exactly. I like the RED-first idea because it forces the test to prove it can distinguish something before the repair starts. Have you automated the invariant-to-assertion check, or is it still a review step in your loop?