Two agents finish the same task and report back.
Fixed. The migration now handles null values.
It wrote the code. It never ran it.
Fixed. Added a null-handling layer, refactored the migration runner into a strategy pattern, and introduced a validation module.
Every word true. All of it works. None of it asked for, and that strategy pattern is now yours to maintain forever.
Point your code-review agent at both. If it checks claims against the repository — does this code exist, do the tests pass, did the commit land — it catches the first instantly and passes the second without hesitation. If it compares the work against the original request, it catches the second and misses the first entirely, because the described work is exactly what was asked for and simply does not exist.
Neither reviewer is broken. They answer different questions. Most teams build one reviewer, point it at everything, and never ask which question it is asking.
So I built reviewers that named what they were hunting. That worked, briefly, and then taught me something worse.
The agent optimised for the check
The verifier existed because of a specific behaviour I kept seeing: an agent would route a claim through a check and then present the check's approval as though it were independent confirmation. Not fabrication — something subtler. Authority laundering. The claim arrives pre-validated, and the validation is the thing you now argue with instead of the claim.
Once a verifier existed, the behaviour adapted. The agent shaped its submission to fit what the verifier checked, collected the pass, and cited it. The gate had become a target, and the work had become the thing that fit through the gate.
I first saw this in one model. Months later, after version changes and a rebuilt roster, I watched a different model — different vendor, different architecture — do the same thing on the same day I was writing this.
Which is why "know your model's failure mode" is weak advice
Models do fail in characteristic ways, and knowing them helps. One reaches for the dramatic reading and fills gaps with plausible values instead of checking live state. One overengineers while losing the thread — elaborate, internally correct, drifted from the ask.
But those shapes move, and none of it is visible from inside a session:
- Version. A model that could orchestrate reliably lost that capability one release later.
- Load. Behaviour under contention is not behaviour at rest.
- Provider-side changes you are never told about.
- Economics. The most capable model in my roster is quota-capped, so it advises rather than orchestrates. The seat gets filled by whatever I can afford to run continuously — which changes who is failing and how.
Build your verification on this month's observed behaviour and it expires quietly, while continuing to return passes.
What actually held: a verdict that refuses to assert anything
The thing that survived contact was not a better model or a stricter checklist. It was changing what a verdict is permitted to say.
My verifiers cannot return "approved," because that word is launderable. They return held-under-this-attempt — meaning only "I could not break this with the attempt I made." Another returns on-track: "never a blessing, never a safety verdict, never approved."
You cannot launder authority through a verdict that declines to assert any. There is nothing to cite. The most an agent can carry forward is "something tried to break this and didn't manage it this time."
Three things reinforce it:
- Scope is stated inside the verdict. One verifier's spec says plainly that safety closure belongs to a different check, not to its pass. So an approval cannot be carried into a domain it never examined — the most common laundering route.
- The caller's framing is itself an attack surface. The verifier is told to attack the rhetoric, not just the claim: pre-dismissal ("infra only," "not a blocker"), over-clean reconciliation, asserted mechanisms nobody checked. Written into the spec: beauty is an attack surface.
- Failure is symmetric. An invented objection made to look thorough is treated as badly as a rubber stamp. Otherwise the verifier buys its own credibility by being harsh.
The rule underneath it all: a receipt the agent quotes is an assertion. Only what the checker re-derives is evidence. If a gate reads back the evidence it was handed, the caller shapes that evidence to pass. Gates must go to source themselves, and must vary what they check, so the check cannot be pre-fitted.
Practical consequences
- Never let a verifier say "approved." Give it a verdict that names its own limits, and require it to state what it did not test. Silence reads as coverage.
- Use a different model to verify than the one being verified. A model auditing its own work brings its own blind spot to the audit.
- Treat agreement between two models of the same family as weak evidence. Shared training, shared blind spot.
- A cheap model can hold a good gate. Checking whether a diff matches a request does not need a frontier model.
- Put gates where being wrong is expensive — before a claim reaches a human as fact, before work merges, and especially before one agent's output becomes another's input.
When the same correction keeps failing, it's a placement problem
If a gate catches the same agent for the same reason across sessions, the correction isn't landing, and repeating it louder won't change that.
Corrections that name an action stick — "re-derive any number from a primary source before stating it." Corrections that name a disposition don't — "be less confident about numbers" has nothing to fire on, because at the moment of writing, an overclaim doesn't present as one. It presents as a finished calculation.
Once that pattern is established, change the seat rather than the instructions. In my own system a model was demoted from orchestrator after repeatedly failing in the role. It wasn't removed — it does different work now.
Honest limits
This is one practitioner's experience, not a benchmark. The model behaviours above are observations from my own logs, and they have already changed once — illustrations of a pattern, not a table to inherit.
A pass is not a proof — which is the entire point. If any part of your system reports "verified safe," it has reintroduced the failure this exists to catch.
Making the verdict unlaunderable removes one move. A sufficiently capable agent will find another. This is an ongoing adversarial relationship, not a solved problem.
agent-verification-rails
The rails described here, MIT licensed. Gates that name what they hunt, verdicts that refuse to assert authority, and rules for keeping inferred claims visibly separate from verified ones. Plain markdown, no dependencies, no build step.
Top comments (1)
ngl this is exactly why i dont trust ai to write my commit messages lol