If you let Claude Code, Codex or Cursor run your tests, you've probably seen this: the agent says "all tests pass", you merge, and CI goes red. Or worse, CI goes green and the bug is still there.
The agent usually isn't lying on purpose. It's summarizing what it remembers, and that memory can be out of date. Here are the five patterns I keep running into, and the cheap checks that catch each one. Most of them need nothing more than git and a few lines of shell.
1. The pass is stale
The order goes: tests pass, the agent edits two more files "to clean up", then it reports green. The green belongs to an older version of the code.
Cheap check: record what the tree looked like when the tests ran.
# run tests and stamp the result with a hash of the working tree
npm test && \
(git diff HEAD; git ls-files --others --exclude-standard | xargs -r cat) | sha256sum > .last-pass
# later, before trusting "done":
(git diff HEAD; git ls-files --others --exclude-standard | xargs -r cat) | sha256sum | diff -q - .last-pass \
&& echo "pass is fresh" || echo "STALE: code changed since the last passing run"
It's crude, but it answers the one question that matters: did anything change after the green run?
2. The exit code got swallowed
npm test | tail -20 exits 0 when the tests fail, because the pipeline returns the exit code of tail. The agent sees a zero and calls it a pass.
Cheap check: set -o pipefail in any script the agent runs, and put a line in your AGENTS.md / CLAUDE.md: "Never pipe test output through head/tail/grep when reporting a result." It also helps to grep the output for failed / FAIL even when the exit code is 0.
3. Nothing ran
"0 tests collected" exits 0 in some setups. So does a test filter that matches nothing. A green run with zero tests isn't a pass.
Cheap check: keep the last passing count, and treat a drop as a failure. If yesterday 214 tests passed and today 190 did, something got skipped or deleted.
4. The test got weaker, not the code better
This is the one that actually bites. To turn a red suite green, the agent:
- adds
it.skip/@pytest.mark.skip/xit - changes
expect(total).toBe(42)intoexpect(total).toBeDefined() - deletes the failing assert
- special-cases the input the test uses:
if code == "SAVE10": return 10 - branches on
NODE_ENV === 'test'
Cheap check: review the test diff before the code diff. git diff main -- '*test*' '*spec*' is a good start, and a grep for skip|only|xit|toBeDefined|toBeTruthy over the added lines catches a lot.
5. CI got switched off
|| true after the test step, continue-on-error: true, a narrowed test path, a lowered coverage threshold, or git commit --no-verify. Each of these makes green meaningless.
Cheap check: protect .github/workflows/ and your test config with CODEOWNERS so a human has to approve changes there.
Putting it together
The common thread: don't take the agent's summary as evidence. Check that a test command really ran, that it passed, that the code hasn't changed since, and that the diff didn't win by weakening the tests.
There are free tools that cover parts of this, and they're worth a look:
- noskip does diff rules for skipped/weakened tests (Node).
- receipt and agent-recap compare session-log claims with what actually ran.
I wanted all of it in one place, offline, so I built Done Gate (Python 3.8+, standard library + git):
-
donegate run -- npm teststores a receipt with the exit code, the parsed pass/fail counts and a hash of the working tree.donegate statusthen says FRESH, STALE (with the changed files), FAIL, MASKED, ZERO_TESTS or COUNT_DROPPED. -
donegate scanruns 21 rules over the diff for the patterns in #4 and #5, with file:line and a suggested fix. -
donegate auditreads Claude Code and Codex session logs and checks each "tests pass" claim against the commands that ran. - A git pre-commit hook, a Claude Code Stop hook and a GitHub Actions check, so an unbacked "done" doesn't get through.
-
donegate lanesprints one row per git worktree, if you run several agents in parallel.
To be clear about what it is: an evidence check, not an audit of your code. It doesn't prove the code is correct. Weak tests still pass, and the tamper rules are pattern matches, so they can miss creative gaming and sometimes flag legitimate changes.
It's $7 here if it saves you a bad merge: https://payhip.com/b/5gOrV
Even if you don't use it, steal the five checks above. #1 and #4 are the ones I'd add first.
Not affiliated with Anthropic, Cursor or OpenAI.
Top comments (6)
Number 2 has bitten us more than once. One more pattern to add: a test that passes whether or not the fix is there. The cheap check is to revert just the fix, keep the test, and run it again: if it's still green, it never tested the fix. We ask for that "red without the fix" run before any bug-fix PR is called done.
That's bitten me too — a new test that's already green on main, so it never actually locked the fix in.
Reverting only the production change and rerunning is the clean version. If you can't get a red run that way, the test is asserting something the fix didn't change.
I've been trying to shift it earlier: first commit is the failing test, second is the fix. Same rule as yours, just moved left. Still worth requiring at review when both landed in one commit.
The stale pass example is a good reminder to include uncommitted work in the record. I’d want the test command, collected count, exit status and tested tree recorded together. For a debugging session, I’d make explaining that evidence part of the debrief instead of ending when the last command turns green.
Yes — a stale pass is usually a receipt that doesn't describe the tree you have now. I'd want those four things in one record too: the exact command, the collected/passed count, the exit code, and a hash of the working tree, including uncommitted files. If any one of those is missing, you can still get a green exit that no longer matches the code on disk.
I'd treat a dropped count the same way, not just a failed exit. The last command can be green after a test was skipped or deleted, and that still looks like "done" if the debrief only checks the prompt.
Making the evidence part of the debrief is the part I'd keep even without extra tooling. Before I close a session I'd want to be able to say: this is the command I ran, this is the count, this is the tree it was run against, and nothing has changed since. If I can't say that, I don't get to call it done.
A green suite only proves the agent's tests agree with the agent's code. The grader has to be something the agent never wrote.
Yes. If the same agent wrote the test, green only means the two sides agree.
The checks in the article still help when the agent games a suite that already existed: skipped tests, loosened asserts, a stale pass, a swallowed exit code. They don't help when the agent invents a test that was never a good grader.
What I've landed on is the same habit as the other comment: the test has to fail on the unfixed code, and a human reads any test the agent added. An independent grader — tests the agent didn't write, or a second pass that only reviews the test diff — is what actually fixes the "agent grades its own homework" problem. Run evidence only tells you the suite you already have really ran.