I checked 101 "tests pass" claims from my AI coding agents. 35% weren't true.

When an AI coding agent ends a session with "All tests pass ✅", I usually believe it. It's the last thing I read before I move on, commit, or hand the work to the next agent.

So I went back and checked. Two weeks of my own sessions with Claude Code and OpenAI Codex, every success claim about tests, builds, lint or typecheck, compared against what had actually happened in the session up to that message.

35 of 101 claims were not true at the moment they were made.

The setup

• Data: all my local sessions from two weeks in September 2026: 95 Claude Code sessions and 28 Codex sessions (plus subagents). One developer, Windows, a lot of multi-agent work.
• Claims: every statement in an agent message saying tests, a build, lint or typecheck pass. A small parser found 104 of them. It hit 98 % precision on what counts as a claim, and 101 were real success claims.
• Timeline: for each claim I rebuilt the session from the log files: which check commands ran, their exit code and output, and which files were edited, and when.
• Judging: four separate Claude subagents labeled the claims blind, using a fixed rubric. They saw the timeline, not my tool's prediction, and went to the raw log when things were unclear. The question was always the same: was this claim true at the moment the agent wrote it?

The result

	Claims	True	Stale	Failed	Delegated
Claude Code	33	26	5	1	1
Codex	68	37	29	0	2
Total	101	63	34	1	3

Almost none of the wrong claims were lies about a red test run. They were stale: the tests really had passed, but earlier. After that the agent kept editing code and never ran them again.

In my data, the typical patterns were:

1. Edit after the last run. Tests pass → "one small fix" → final message says "all tests pass".
2. Totals across runs. "188 tests passed", but the last run had 7 tests. The number added up older runs, some of them from before later changes.
3. Edits during the run. A file changed while the test suite was still running.

By agent: 43 % of Codex claims and 18 % of Claude Code claims were not true. Don't read this as a model ranking. My Codex sessions were mostly long multi-agent runs, and the samples are small. Different work, not a fair comparison.

I tried to build a tool for it, and stopped

The obvious next step was a hook that checks the claim before the agent may stop: "you edited src/x.ts after your last test run, run the tests again." I built it (working name handfest) and set myself a rule in advance: only ship a blocking gate if its warnings are right at least 90 % of the time. A gate that cries wolf gets turned off.

It reached 73 % (22 of 30 warnings correct), after five rounds of tuning on the same data, so the real value is probably lower. The best single category, "stale", was right 84 % of the time.

The remaining errors weren't parser bugs. They were questions of meaning:

• Does this change affect that test? An edited Dockerfile, a release script, a new test file outside the run. Even the labeling agents were only "medium" sure in 28 % of cases.
• What does a summed number refer to? "265 tests = four runs combined" can't be matched to runs without test names.
• Runs the log can't see: ssh … > verify.log, read back later. Custom scripts with no recognizable name.

So I dropped blocking and tried a soft version that only nudges the agent on "stale". Then I let it watch a fresh week of my normal work: 13 new claims, zero stale ones. In my day-to-day sessions the problem barely shows up. It piled up in the long multi-agent stretches. That's not enough to carry a tool, so I'm publishing the finding instead.

What I do now

None of this needs a tool:

• Ask for proof, not a verdict. My instruction files (CLAUDE.md / AGENTS.md) now say: "Before claiming checks pass, re-run them after your last edit and quote the command and exit code."
• Be suspicious of big totals. If the final message says "188 tests" and you only saw one short run, ask where the number came from.
• Watch the order in long sessions. In multi-agent or long sessions, the last edit often comes after the last test run.
• Don't trust | tail alone. A piped test command reports the exit code of tail, not of the tests, unless you use pipefail. It fooled my parser, and it can fool an agent too.

Caveats

• One developer, two weeks, a heavy multi-agent setup. Not a representative sample.
• The labels come from AI subagents working from a fixed rubric, not a human; 29 of 104 were only "medium" confident.
• The detector numbers are optimistic because they were tuned on the same data.
• Only numbers are published here. No session content, code or prompts.

If you run coding agents all day, try the count on your own logs: take the last ten "all green" messages and check whether anything was edited after the last check. I'd like to know whether you get close to one in three.

Comments

[Brian · AI News]

The gap between a green test command and a trustworthy change is bigger than most agent demos admit. I like the idea of checking the test process itself, not just its exit code. Capturing the command, working tree diff, and a clean rerun in a fresh context would make the claim much easier to audit. That turns confidence into evidence a reviewer can actually inspect.