I checked 101 "tests pass" claims from my AI coding agents. 35% weren't true.

When an AI coding agent ends a session with "All tests pass ✅", I usually believe it. It's the last thing I read before I move on, commit, or hand the work to the next agent.

So I went back and checked. Two weeks of my own sessions with Claude Code and OpenAI Codex, every success claim about tests, builds, lint or typecheck, compared against what had actually happened in the session up to that message.

35 of 101 claims were not true at the moment they were made.

The setup

• Data: all my local sessions from two weeks in September 2026: 95 Claude Code sessions and 28 Codex sessions (plus subagents). One developer, Windows, a lot of multi-agent work.
• Claims: every statement in an agent message saying tests, a build, lint or typecheck pass. A small parser found 104 of them. It hit 98 % precision on what counts as a claim, and 101 were real success claims.
• Timeline: for each claim I rebuilt the session from the log files: which check commands ran, their exit code and output, and which files were edited, and when.
• Judging: four separate Claude subagents labeled the claims blind, using a fixed rubric. They saw the timeline, not my tool's prediction, and went to the raw log when things were unclear. The question was always the same: was this claim true at the moment the agent wrote it?

The result

	Claims	True	Stale	Failed	Delegated
Claude Code	33	26	5	1	1
Codex	68	37	29	0	2
Total	101	63	34	1	3

Almost none of the wrong claims were lies about a red test run. They were stale: the tests really had passed, but earlier. After that the agent kept editing code and never ran them again.

In my data, the typical patterns were:

1. Edit after the last run. Tests pass → "one small fix" → final message says "all tests pass".
2. Totals across runs. "188 tests passed", but the last run had 7 tests. The number added up older runs, some of them from before later changes.
3. Edits during the run. A file changed while the test suite was still running.

By agent: 43 % of Codex claims and 18 % of Claude Code claims were not true. Don't read this as a model ranking. My Codex sessions were mostly long multi-agent runs, and the samples are small. Different work, not a fair comparison.

I tried to build a tool for it, and stopped

The obvious next step was a hook that checks the claim before the agent may stop: "you edited src/x.ts after your last test run, run the tests again." I built it (working name handfest) and set myself a rule in advance: only ship a blocking gate if its warnings are right at least 90 % of the time. A gate that cries wolf gets turned off.

It reached 73 % (22 of 30 warnings correct), after five rounds of tuning on the same data, so the real value is probably lower. The best single category, "stale", was right 84 % of the time.

The remaining errors weren't parser bugs. They were questions of meaning:

• Does this change affect that test? An edited Dockerfile, a release script, a new test file outside the run. Even the labeling agents were only "medium" sure in 28 % of cases.
• What does a summed number refer to? "265 tests = four runs combined" can't be matched to runs without test names.
• Runs the log can't see: ssh … > verify.log, read back later. Custom scripts with no recognizable name.

So I dropped blocking and tried a soft version that only nudges the agent on "stale". Then I let it watch a fresh week of my normal work: 13 new claims, zero stale ones. In my day-to-day sessions the problem barely shows up. It piled up in the long multi-agent stretches. That's not enough to carry a tool, so I'm publishing the finding instead.

What I do now

None of this needs a tool:

• Ask for proof, not a verdict. My instruction files (CLAUDE.md / AGENTS.md) now say: "Before claiming checks pass, re-run them after your last edit and quote the command and exit code."
• Be suspicious of big totals. If the final message says "188 tests" and you only saw one short run, ask where the number came from.
• Watch the order in long sessions. In multi-agent or long sessions, the last edit often comes after the last test run.
• Don't trust | tail alone. A piped test command reports the exit code of tail, not of the tests, unless you use pipefail. It fooled my parser, and it can fool an agent too.

Caveats

• One developer, two weeks, a heavy multi-agent setup. Not a representative sample.
• The labels come from AI subagents working from a fixed rubric, not a human; 29 of 104 were only "medium" confident.
• The detector numbers are optimistic because they were tuned on the same data.
• Only numbers are published here. No session content, code or prompts.

If you run coding agents all day, try the count on your own logs: take the last ten "all green" messages and check whether anything was edited after the last check. I'd like to know whether you get close to one in three.

Comments

[Reid Marlow]

The post-pass cleanup edit is where this bites us almost every time. An agent gets a green test run, notices an unused import or a debug print, removes it, and immediately wraps up the turn without re-running the suite. The fix that finally held up in our harness was binding verification to git mtime rather than parsing the agent's prose. If any tracked file in the worktree has a modification timestamp newer than the stdout capture of the test command, the harness rejects the completion call and forces one final clean execution. It catches the stale claims without having to guess intent.

[Eusebiu Balan]

What changed it for me was never reading the summary line. If the raw output of the command is not in the transcript I treat the claim as unverified, which sounds paranoid until you have merged one of those 35.

[Maksym Danylenko]

Thanks for counting instead of guessing. The "edit after the last run" pattern is the one we trip over most too. We keep hitting the same gap one level down: even a true "all tests pass" doesn't tell you whether those tests would fail on a bug. Mutation testing is the only check we trust for that.

[rulereceipt]

This is the exact problem that got me building something. The 35% matches what I saw in my own sessions.

I took a narrower route than checking whether the code is actually correct. My tool (rulereceipt, I built it) reads the session log and looks for proof. If the agent says "all tests pass" but no test command ran in that session after its last edit, it flags that claim and quotes the line. The limit is that it can't see a test you ran yourself in another terminal, so it tells you exactly what it looked for instead of pretending to know more.

I'd be curious what made your version hard to trust. For me, the hard part was never catching the problem, it was not flagging things that were actually fine.

[Brian · AI News]

The gap between a green test command and a trustworthy change is bigger than most agent demos admit. I like the idea of checking the test process itself, not just its exit code. Capturing the command, working tree diff, and a clean rerun in a fresh context would make the claim much easier to audit. That turns confidence into evidence a reviewer can actually inspect.