This is a republish from my site; the original, with the full reviewer files, is at codebyramiz.com.
You have an AI coding agent's report in front of you, and it says the tests pass. What can you conclude from that? Less than it feels like. Here is a case from my own ChatPing iOS app.
A change came back with 768 passing tests reported. I sent it to an independent reviewer. The reviewer did not rerun those tests, and its verdict begins VERDICT: FAIL, with the first two findings rated High:
[...] concurrent opens can mix chat identity and transcript. [...] If opens for A and B interleave, A can bind the ChatCapability to A while copying B's SDK snapshot into
rawConversation. [...] The new tests cover only one sequential open.[...] reopening a failed reply changes recency and rewrites the file. [...] The test fixture contains only a completed reply.
The report said 768 passed. The reviewer found two cases those tests did not cover. A green count says how many checks ran, not whether they were the right ones. The fix took two more review rounds before the same reviewer passed it.
This piece is about the three checks I put between an AI coding agent and the word "done", with the real files and verdicts. It's the same setup as how I review AI changes in a production mobile app, seen from the angle of what counts as proof.
Why an agent that says the tests pass is only making a claim
The agent that wrote the code also chose which tests to write, ran them, read the output and summarised it for me. Every step of that chain is the same author. If it misses a case when writing the code, it will often miss the same case when writing the test, and then report green in good faith.
Agents also get asked what is happening right now. "Is the gate still running?" is a claim about a process, and a model can answer it from whatever text is in front of it instead of from the process table. The seven-hour hung test run at the start of running several Claude Code agents on one machine was a version of that: a log said one thing, the machine said another.
So a report from an agent is a claim, and a claim gets checked by something that did not make it. What does that check look like when the claim is about code, about status and about a run that went quiet? Three versions follow.
A reviewer that refuses to pass tests it could not run
The reviewer in these examples is Codex, run read-only through a wrapper that demands a last line of exactly VERDICT: PASS or VERDICT: FAIL. A different worker writes the code; the worker is never the gate.
The case I find most useful is not a catch. It is a module in my shared ping-ai-sdk kit, a port of a grounding renderer from Android to iOS. The review request asked the reviewer to build and run the tests itself, so there would be pass counts that did not come from the author. It read the diff and compared it with the Android version, and it found no blocking issues in static review. It could have written "looks right, probably passes". It couldn't run the tests, so there were no independent counts, and it failed the module on that alone:
No blocking code/file:line issues found in static review. [...]
Gate blocker: I attempted all requested SwiftPM commands from `ios/`, but this
session's read-only sandbox prevented Swift/Xcode from creating `/tmp/xcrun_db-*`
cache files, so no independent pass counts were produced:
- `swift build` failed before compilation: `errno=Operation not permitted`
- `swift test --filter GroundingRendererTests` failed same way
- `swift test --filter TextFeaturesTests` failed same way
- `swift test` failed same way
VERDICT: FAIL
Does the agent's "still running" claim have a fresh check behind it?
The second check is about status, not code. When an agent tells me a build or a device run is "still in progress", I want that to rest on something it just looked up.
A CLAUDE.md rule asking for that did not hold, so it became a Claude Code Stop hook, verify-before-status-claim-hook.sh. It runs when the agent tries to end its turn and reads the transcript for that turn. If the assistant text matches a status-claim pattern and the turn contains no qualifying check, it blocks the stop. This is the pattern, abridged and not runnable as shown:
# abridged: [...] marks removed alternatives
claim_re='waiting for|still waiting|in progress|currently (mid|waiting|[...])|
should (complete|finish|fire|be done)|probably (done|finished|complete)|[...]|
still (running|underway|executing|active|going|working)|[...]|
(job|task|process|work|fork|subagent|it) (is|remains) (active|running|[...])'
A qualifying check is a Read of an image file, or a Bash command whose leading token is ps, adb, stat, date, logcat and a few more. When the hook applies, the agent gets this back and the turn does not end:
BLOCKED: this reply states or implies async/background work is "in progress" / "still
running" / "waiting" [...] but this turn contains no fresh verification [...] If you
already believe it's true from memory or a prior turn, that is exactly the failure mode
this hook exists to catch [...]
The second guard is the part people don't expect. Even when the claim is backed by a real ps, the hook blocks unless the turn also schedules the next check. One verified look is a snapshot, not monitoring. So the agent has to book a wake-up within 15 minutes, and the wake-up has to name something concrete to inspect, such as a process, a device or a log's modification time. "Check back later" doesn't count.
I should be plain about what this is: a heuristic. It checks that the current turn contains a qualifying tool call, and a bare date counts. It does not check that the call was about the task being claimed. It also stands down in some cases: when the turn itself just started background work (the wake-up guard below covers that turn instead), when jq or the transcript is missing, when the hook is already re-running after a block, and in headless worker sessions that have no next turn. It narrows the gap between "still running" and a fresh ps; it does not close it.
The script itself isn't published; it is Bash plus jq, reads the turn from the transcript path Claude Code passes on stdin, and the wake-up guard looks for a ScheduleWakeup tool call in the turn. It is registered as a Stop hook in Claude Code's settings.json:
"hooks": {
"Stop": [
{ "hooks": [ { "type": "command",
"command": "$HOME/.claude-shared/scripts/verify-before-status-claim-hook.sh" } ] }
]
}
Seven review rounds on one hook
I did not write that hook correctly the first time. The file's own header is the changelog, and it reads like a list of ways to be wrong:
| Version | What the reviewer found |
|---|---|
| v2 |
tool_result entries were logged as user turns, so the turn boundary was wrong and evidence was discarded |
| v3 | the word date anywhere in a command, even printf date, counted as a live check |
| v4 | a wake-up prompt of "check back later" passed with no concrete target |
| v5 | one evidenced "still running" and then silence: monitoring ended after a single recheck |
| v6 | a wake-up whose tool call had errored was still counted as scheduled |
| v7 | "The background process is running" matched none of the claim phrases |
| v8 | the phrase list kept missing wordings, so it became a subject-plus-state pattern |
Those were Codex adversarial re-reviews: seven rounds against the same hook on one day. Every finding was real. My CLAUDE.md says a reviewer's finding is evidence, not an assignment, and that after two rounds on one target a third needs a written decision: fix it, register it as a follow-up, or delete the check. I didn't stop at two, and I'm not going to claim the later rounds were each priced. Whether rounds three to seven were worth it, I can't tell you. Round 7 in particular was another version of the same problem one level up, which is the pattern that rule warns about.
The later versions, v9 and v10, were not about correctness. Before v9 the hook re-read the whole transcript on every turn, which on a 71,525-line session cost about 9.5 minutes per turn. After v10 pulled the tool-call fields out in a single jq pass, a 200-call turn took 94 ms where it had taken 3.5 s. A check that takes minutes gets turned off.
Exit codes that separate "failed" from "no result"
The third check is for long runs. A release gate, a device suite or a build can go quiet. If a hung run gets reported as a failing test, I go hunting for a product bug that doesn't exist, and if it reads as "nothing wrong yet", I wait.
Every run expected to take more than a few minutes goes through a watchdog wrapper, and its exit codes carry the distinction:
| Exit code | Meaning |
|---|---|
| 0 | the command succeeded |
| 125 | stalled: no output for the stall window; diagnostics attempted, process tree killed |
| 124 | hard cap exceeded; diagnostics attempted, process tree killed |
| 143 | the watchdog itself was signalled (TERM, INT or HUP); it killed the tree and exited |
| 2 | usage error |
| other | the command's own exit code, passed through |
The line in long-run-watchdog.md that matters is this one: 124 and 125 are not product verdicts. They mean the run produced no verdict. One catch: a child process that exits 124 or 125 on its own looks identical from outside, so the watchdog prints a stderr line saying which one it was. A stalled run is classified from its diagnostics first, and only then is it a test failure, a tool bug or nothing.
The review gate follows the same logic. Exit 0 means a verdict exists, and I read the final verdict line (VERDICT: PASS or VERDICT: FAIL) and the opening banner, which says whether Codex or the Opus substitute produced it. A timeout is "inconclusive, not a pass", and so is any other non-zero exit. When Codex is out of quota, the wrapper falls back to a Claude Opus adversarial review, and that fallback has to produce a verdict too. No verdict never rounds up to approval.
What to copy from this setup
- Make "done" a line a machine can parse (
VERDICT: PASS), produced by something that did not write the code, and treat a missing line as a fail. - Make the reviewer run the tests itself, and let it fail the work when it can't.
- Keep exit codes honest. "I don't know" and "it failed" must not share a number.
- If you write a Stop hook for status claims, budget your review rounds before you start. Mine got seven in a day.
None of this makes the code correct. A reviewer can be wrong, and a regex can be fooled. What it does is move the question from "do I believe the agent" to "what did a second party see", and that one I can answer without reading the diff. For how the code itself stays clean when nobody reads it, see what keeps agent-written code clean.
Originally published at codebyramiz.com.
Top comments (0)