The harness looks clean on paper. An agent gets a task, checks out a branch, edits the repo, spends its command budget, and reports what it changed. The harness ignores the report. It runs the test suite itself and inspects the diff itself. Then a separate model reviews the patch against a list of invariants: authorization checks stay enforced, input validation is preserved, exceptions are not silently swallowed, the public API contract does not drift.
For each invariant the reviewer returns a verdict of held, violated, or unverifiable, and each verdict carries a file:line citation. The merge contract is mechanical. Every invariant must be addressed, every held or violated claim needs evidence, any violated invariant blocks the merge, and a fully clean verdict clears the change for a squash merge.
That architecture deserves credit, because it has the right instinct. The implementer is treated as unreliable. Its prose summary carries no authority. "Tests passed" means nothing until the harness runs the tests, and "I preserved behavior" means nothing until another process checks the change. Trust moves out of the agent's mouth and into repeatable checks. Real progress.
Then the distrust stops. The reviewer says every invariant held, the JSON parses, each row has a citation, and the contract accepts it. But the reviewer has just produced a self-report about its own review work, and the harness swallows that self-report whole.
That is the asymmetry. The coding agent is distrusted because it can be confidently wrong. The reviewer can be confidently wrong in exactly the same way. Only one of those two errors usually gets measured.
Look at what a merge contract actually checks. It checks shape. Did the reviewer address every invariant, use the expected schema, cite a line, avoid emitting "violated"? Those checks are cheap and deterministic, and they earn their place: they catch empty reviews, broken prompts, parser failures, and lazy outputs that skip half the policy. Shape is worth enforcing. Shape is also weak. A reviewer that returns "held" for everything, with plausible citations attached, sails through that contract forever. It can read nothing and catch nothing and still satisfy the gate. The contract sees a complete artifact. It never sees whether the artifact is correct.
That is the failure mode worth naming: completeness without correctness. The output looks serious. Each invariant gets its paragraph, each paragraph points at code, the verdicts are green, the structure is valid, the CI step is happy. Nothing along that path shows the reviewer would have caught a real violation. A citation can be decorative, pointing near the relevant function while missing the branch that dropped the guard, or naming the call site while ignoring the callee that actually changed. The review can be complete and false at the same time.
This bites harder in autonomous coding because the merge gate tends to bundle several weak signals and present them as one strong one. Tests pass, static checks pass, the reviewer reports the invariants held, the final line goes green. Those checks cover different surfaces, though. Tests cover the examples that happen to execute. Static analysis covers known syntactic and type-level patterns. The reviewer is usually handed the gap between them: semantic regressions, policy invariants, the "this must never happen" constraints that nobody encoded as a test. That gap is where reviewer recall becomes a quality gate.
Recall is the plain question: of the violations actually present, what fraction did the reviewer mark as violated? Plant ten known violations, catch four, and recall is 0.4 for that set. A reviewer at 0.4 can still produce gorgeous review text, satisfy every schema rule, and cite real lines while missing most of the bad changes it exists to find. Precision matters too, especially when false alarms block good work, but precision announces itself, because a blocked merge creates visible friction that people feel. Misses are silent. A missed violation merges cleanly and turns into someone's incident three weeks later. The dangerous number is the one nobody sees.
So measure it. Mutation-test the reviewer. Feed the review path diffs you know are bad and record whether it detects them. Take a scratch worktree from a real repository state, apply a small mutation that breaks exactly one invariant, run the same reviewer used in the merge gate, and score the result. If the invariant is that authorization stays enforced, delete one auth check on a sensitive endpoint. If it is that a null input keeps its existing error contract, drop the null check. If it is stable pagination, change <= to < at the boundary. If it is that failed writes are never reported as success, swallow the exception and return a success value. Each mutation has an expected violation, and the reviewer either flags it or misses it. Now the gate carries a measured recall instead of an assumed one.
Score it per invariant, because an aggregate hides the exact weakness that matters. One reviewer catches obvious validation removals but sleeps through authorization drift. Another handles local diffs and falls apart when the invariant spans two files. A respectable overall number can sit on top of a critical class scoring near zero. And run the mutations through the production path: same prompt, same context budget, same output contract, same parser. A special evaluation prompt measures a reviewer you do not ship. If production permits "unverifiable," count it deliberately, because a known violation returned as "unverifiable" did not protect the merge.
That verdict is its own hazard. "Unverifiable" exists for honest reasons. Some invariants genuinely cannot be judged from the diff alone when runtime config, generated code, flag state, or an out-of-context contract file is missing, and forcing held or violated there manufactures fake certainty. But it doubles as a hiding place. A weak reviewer routes hard cases into it. A degraded reviewer routes nearly everything into it. A prompt that asks for careful humility drifts into routine abstention. If the contract reads unverifiable as neutral, the reviewer can quietly stop reviewing while the gate keeps passing. Track the rate. Two percent may be fine for a narrow, well-contextualized invariant set. Forty-five percent means the reviewer has stopped making decisions. A jump after a prompt edit, a context cut, or a model swap is a regression, and it reads clearly when you slice the rate by invariant. "Cannot judge dependency license impact from this diff" is defensible. "Cannot verify whether auth checks were preserved," with the auth files sitting right there in context, is the reviewer going dark on the thing you most needed it awake for.
None of this needs a giant benchmark. Twenty mutations across the invariants you care about teach more than a thousand clean reviews. Store each one as a patch with its metadata: the invariant, the mutation, the expected verdict, and a detection rule that demands the reviewer name the right invariant and point at the changed code, rather than just emitting the word "violated" somewhere in its output. Keep the merge-facing score blunt: did it catch the planted violation, yes or no. A recall number that leans on generous interpretation just becomes another self-report. Mix the layers so the estimate stays honest: a few blatant mutations, a few subtle local ones like an inverted flag in a fallback path, and a few that cross a boundary so the handler still looks fine while the invariant breaks underneath it. Past regressions from your own history make the best mutations, since they encode failure shapes the code actually produces. Include clean controls too, unmutated diffs that should come back held, or a reviewer that screams "violated" at everything will post a great recall number while being useless as a gate.
Once recall is a measured quantity, the contract can finally be honest about what it trusts. A reviewer at 0.9 on the critical suite earns more weight than one with unknown recall and elegant prose. A reviewer at 0.35 is decorative for those invariants no matter how clean its verdicts look, and a reviewer whose unverifiable rate doubles overnight should be pulled off the important gates until someone understands why. The number is not universal. It belongs to a repo, an invariant set, a context strategy, and a specific prompt, and it moves when any of those move. That is the point. You stop pretending the reviewer is a constant.
The underlying principle is simple. A verifier is software even when a model writes the verdict, with inputs and blind spots and regressions like any other component, and its output format submits to ordinary validation while its detection ability only reveals itself under known-bad inputs. A harness that distrusts the implementer and trusts the reviewer for free has just moved the unexamined assumption one step downstream. The author says "I did the work." The reviewer says "I checked the work." The second sentence can be exactly as wrong as the first. Feed the verifier a violation on purpose and see whether it notices.
Top comments (0)