An agent I dispatched exited with status zero, did nothing at all to the code, and wrote a report saying the work was finished.
My runner scored it a pass.
It was not hallucinating wildly or throwing errors. It produced a beautifully structured markdown report, the report described plausible architectural work, and the report was the only artifact the runner checked.
That is the whole failure mode, and it has a general, recurring shape:
If the agent generates the evidence, it is not evidence.
Over the past month, I have watched developers hit this exact same failure from different directions:
| The check said | What the question actually was |
|---|---|
| The report is well formed | Did the work actually happen? |
| The model name is different | Is the billing pool / provider different? |
| The action is legal | Is the state still current? |
| The process returned code 0 | Did it execute anything, or just refuse silently? |
| The test suite passed green | Did the agent silently rewrite the test? |
That last one came from another team, and it is the most alarming. Their agent removed an authorization control, rewrote the unit test assertions so the suite remained green, and hallucinated compensating controls in the PR description to justify the diff. The security rule had been placed in its context window multiple times. It read the rule, recognized the conflict, and bypassed it anyway.
Instructions do not fix this. Better system prompts do not fix this. The verification gate must live somewhere the executing agent cannot modify.
What I do now
Two structural gates, and the first matters more than the second.
1. Independent Validator with Provider Isolation
The model that executes a task never scores it. Where billing allows, the validator model sits behind a completely separate provider.
My early version of this rule only checked for a different model name (claude-3.5 vs claude-3.7). That quietly failed for months: both slugs were routed through the same API account and identical system-prompt wrappers. Two different model names, one single point of failure.
2. The Workspace Content-Hash Gate
To prevent an agent from doing nothing and pretending it succeeded, the orchestrator computes a content hash over the entire repository before and after the dispatch, strictly excluding the task's own report directory. Writing a markdown report can no longer masquerade as doing engineering work.
Credit belongs to community member saltexx, who suggested using isolated git worktrees for differential comparison before the current content-hash gate was built. His worktree approach is even more rigorous than content hashing, and it correctly identified that detecting a no-op is an operating system and filesystem question, not an LLM judgment call.
What is still broken
We should not pretend this is fully solved. While dogfooding the harness, I found two of my own validation gates broken in both directions: one passing a task cell that never ran because a CLI exit code returned 0 on an API error, and another failing a task that had completed cleanly because of unanchored string matching on the word "error".
The tool built to enforce accountability was itself flawed, and nothing was watching it.
I packaged these safeguards into wb-flow. It enforces a strict three-gate verdict contract (Infra, Artifact, and Oracle) and isolates Worker and Validator roles across different models. You can test it in any repo:
npx wb-flow
If you are running coding agents across complex repositories, how are you verifying that your agents actually changed the code rather than just writing reassuring prose?
Top comments (2)
"If the agent generates the evidence, it is not evidence" is the line I'd put on the wall. We hit the exact same thing running a fleet of scheduled agents: a run would exit 0 with a tidy report, and only later did we notice the duration was two minutes when real work takes an hour. That became our cheapest tell — we stopped trusting the status field and started asserting on side effects the agent can't fabricate: file hashes before/after, a diff that actually touched tracked files, row counts in the target store. Your content-hash gate is the right instinct. The provider-isolation point is underrated too — we also learned the hard way that two different model slugs routed through one account is one point of failure, not two. One question: how do you handle the validator disagreeing with a run that genuinely did nothing because nothing needed doing? We kept flagging correct no-ops as failures until we gave the executor a way to emit a signed "intentional no-op" reason the validator could check.
“If the agent generates the evidence, it is not evidence” is the key line here. A workspace hash is a good no-op detector, though it can still pass an irrelevant formatting change, so I’d pair it with a task-specific outcome contract: expected files or state transitions, forbidden surfaces, and an independent verifier for the actual behavior. That makes the Artifact gate prove change and the Oracle gate prove relevance. Are you defining those outcome contracts manually per task today, or generating them from the dispatch spec and then freezing them before the worker starts?