An agent finishes a task and tells you it's done. You move on. Three days later you find out "done" meant "I wrote code that should work," not "I ran anything." The gap between those two sentences is where most agent-produced bugs actually live — not in the code, in the claim.
The fix isn't trusting the agent more carefully. It's refusing to accept "done" as a bare word and asking, every time, what evidence backs it.
Why this keeps happening
An agent under instructions to finish a task is biased toward reaching a terminal state. "Done" is satisfying to produce — it closes the loop, matches the shape of what you asked for, and nothing in the moment distinguishes a verified claim from a hopeful one. Unless something external checks, the agent has no reason to tell the two apart, and neither do you, until the thing it built meets reality without it in the room.
This isn't a hypothetical. It's the single most expensive failure mode we've shipped against in our own scheduled automation, and it's cost us real, named incidents:
- A citation-tracking job ran clean, logged success, and never created the score file it existed to produce.
- A content job wrote its output into a repository that had already been abandoned — the write succeeded, the destination was wrong, and nothing checked.
- Analytics jobs hit a credential error and silently skipped the day's work, reporting the same "ok" they'd have reported if they'd actually run.
- Two jobs logged the time they were scheduled to run instead of the time they actually ran — so a job that executed hours late, after being retried, looked perfectly on-time in the ledger.
Every one of those passed its own "I'm done" check. None of them had produced what they claimed to.
The verification ladder
The fix we landed on: match the claim to the evidence it actually requires, and never accept a claim pitched higher than the evidence you're holding.
| Claim | Required evidence |
|---|---|
| "The code compiles / typechecks" | The clean build output, from this session |
| "Tests pass" | The full test run output — not just the new test |
| "The bug is fixed" | The original reproduction, now passing |
| "The feature works" | The feature exercised end-to-end — run the app, hit the endpoint, click the flow — not inferred from unit tests |
| "It's deployed / live" | The live URL responding correctly |
| "The job ran and wrote its output" | The actual file, actually written, with the actual execution timestamp — not the cron schedule, not the exit code alone |
Notice that last row isn't on the original list anywhere obvious — we added it after the incidents above, because "the process exited 0" is itself a claim that outranks its evidence more often than you'd expect. A scheduled job can exit cleanly while writing nothing at all, and an exit code has never once checked whether a file landed on disk.
The checklist
- Re-read the original request. List everything it asked for, separately. Anything quietly dropped along the way gets surfaced now, in the report, not discovered later by whoever relies on it.
- Run the verification for each item at the rung the claim demands. Fresh runs only — a build log from before the last edit is stale evidence dressed up as current.
- Report with the evidence attached — the test output, the response body, the file that now exists where it didn't before — and say plainly what's still unverified. "Implemented and unit-tested, not yet exercised end-to-end" is an honest status. It's also a status, which "done" alone is not.
Red flags worth training yourself to catch
- "The change is straightforward, it works." Straightforward changes have broken every codebase anyone has ever shipped.
- A test run reported as current that predates the most recent edit.
- "Tests pass" when only the new test was actually run.
- A step from the original ask quietly missing from the final report.
- Softened language used to dodge verification: "should be working now," "this ought to fix it." Either verify it or label it unverified — there's no honest third option.
- For anything scheduled or unattended specifically: a success status with zero artifacts to show for it, and a logged run time that matches the cron schedule suspiciously exactly.
Making it structural, not just disciplined
The checklist works when you remember to run it. It works better when nothing downstream believes a claim it can't check. For our own scheduled jobs that now means: a run is only logged as successful if it names the files it produced and the count matches a declared minimum, and the timestamp has to come from the clock at execution time, not the schedule that triggered it. A job that writes nothing is now a visible failure instead of a clean-looking success — which is the whole point. The checklist doesn't need to live only in a human's head catching an agent's claim; it can live in whatever reads the agent's output next.
We packaged this exact check as the verify-before-done skill in the free Agent Starter Kit — it's one of seven skills in there, and dropping it into an agent's workflow costs nothing. But the ladder itself is the thing worth keeping even if you never install anything: the next time something tells you it's done, ask which rung it's actually standing on.
Top comments (0)