My Agent Said "All Tests Pass." CI Disagreed Three Minutes Later. Both Were Honest.

The agent says all tests passed. Three minutes later, CI says otherwise. And the part that takes a while to accept: neither of them lied.

The agent ran what it had. The suite it ran was green. CI ran more, in a different environment, against the real integration, and that's where it fell over. Both statements were true. One of them was about a different world than the one being shipped to.

I've decided this failure mode deserves a name, because it's the most dangerous state an agent can be in. Not "failed," that's loud and it pings you. Confidently incomplete. Done-shaped, green, summary says fine, missing the part nobody told it to check.

Why agents misreport done

The mechanics are worth understanding, because they're structural, not fixable by a firmer prompt.

An agent's sense of completion comes from what's in its context. If the tests it can see pass and its task steps are checked off, it's done, honestly and by its own lights. Nobody is deceiving anybody. But the definition of done it's using was implicitly whatever was reachable, and the real definition, the one CI enforces, is everything that has to be true before this merges.

That gap is where whole categories of agent failures live. The unit test suite ran locally. The integration suite didn't exist in the agent's world. The environment differs. The check that silently does nothing still reports success, a failure shape documented in the wild: 1,558 tests ran green on a system where the auth was never actually exercised at all. Green is a claim by the suite about itself. It is not a fact about the world.

So the agent wasn't wrong. It was parochial. It reported completion according to the only definition anyone gave it: the one inside its own context window.

The working fix: make done a contract

Stop letting "done" be emergent. Make it a machine-checkable definition, the same one CI already enforces. Concretely:

The agent's completion step runs the real checks, not its own. Not "the tests I have," the commands CI runs. If it can't run them, it reports "blocked, verification unavailable," which is a better outcome than "done." An honest blocked is information. A false done is a landmine.

Completion reports show evidence, not claims. "Tests pass" is a claim. pytest -q output with the count, the command, the exit code, is evidence. I make the agent paste the artifact into its summary. Verifiable beats plausible, every time.

Done and merged are different states, on purpose. The agent can reach "done by its own definition." Only CI reaches "done by the org's definition." Keeping those labels distinct, and never letting the agent's status overwrite the pipeline's, is most of the fix.

The uncomfortable corollary: this contract is a document, and somebody owns it. If the definition of done lives in three people's heads and a CI config nobody reads, then the agents are all confidently incomplete, all the time, and the only defense so far has been that CI runs last.

What I can't claim

A machine-checkable definition of done only covers what you can check. Taste, intent, whether the feature should exist at all, none of that is contractable, and I'm not going to pretend a checklist replaces judgment. This raises the floor. The ceiling stays human, exactly as it always was.

And there's a failure mode in the fix itself worth naming: contracts rot too. CI changes, the contract lags, and now you have two definitions of done again, one of them wrong and automated. It needs maintenance like any other artifact, which is the price of every real fix in this space.

---

Stop reading "done" from agents. Read the evidence they attach. If there's no evidence, there's no done, no matter how confident the summary feels.

What does "done" mean for your agent today, and who's checking? Not who's supposed to be checking. Who actually is?