My AI agents merged 2,184 PRs in 7 weeks. 1,072 of them were fixes.
I run a one-person company, and coding agents write most of the code. Last month an audit of my own repo found that a big share of fix PRs were "certified" by nothing more than the agent saying it works. So I added a Claude Code Stop hook that refuses to let the agent end its turn on an unproven claim.
Then I did the part most posts about these hooks skip. I measured it.
What the hook does
A Stop hook runs when the agent tries to finish its turn. Mine reads the agent's final message and applies a few rules:
- If the message claims an outcome ("fixed", "deployed", "live", "verified", "done", "is working") and carries no evidence, the turn is blocked and the agent is told to prove the claim or downgrade it.
- Evidence means command output in a fenced block, a before/after probe, a CI state tied to a PR number, or a passing live check.
- "Green" is banned outright. A PR has a state, not a mood.
- Honest negatives pass: "not deployed yet" is fine.
- Text inside code fences is ignored, so the agent can quote a rule or a log.
- It never blocks twice in a row (the
stop_hook_activeflag), so it cannot trap a session. - If anything goes wrong (missing tool, bad JSON, unreadable transcript), it lets the message through. A gate that wedges your agent gets turned off within a week.
At least six open-source hooks do roughly the same thing. What I could not find was anyone publishing what happened after.
How I measured it
Two raw sources, no dashboards:
- Session transcripts. Claude Code writes every session as JSONL. I parsed all of them for the hook's block message, recorded the text that got blocked and what the agent wrote next, then drew a random sample of 60 blocks and labelled each one: correct block, misfire on quoted or drafted text, misfire on an honest negative, or misfire where evidence was already present.
- Merged PR bodies. About 1,090 merged PRs before the hook (Sep 20 to Oct 14) and 1,090 after (Oct 15 to Nov 7). For each fix PR I checked whether the agent certified its own fix without a failing run, and for every PR whether the bug it fixed was found by a user or by me.
Results
| Before | After | |
|---|---|---|
| Fix PRs self-certified with no failing run | 103 / 483 (21%) | 37 / 576 (6%) |
| PRs fixing a bug a user or I found | 76 / 1,092 (7.0%) | 77 / 1,091 (7.1%) |
The gate changed what the agent says. It did not change what reaches users.
And the gate is noisy:
- 401 blocks across 121 sessions, roughly one every six turns.
- In the sample of 60, 27 blocks were right (45%, 95% CI roughly 32 to 58%).
- 22 misfires hit quoted or drafted text. It blocked drafts of this article four times.
- 8 misfires hit honest "not done yet" status. 3 had evidence already.
- After a block, the agent ran a real check 37 times out of 60, softened the claim 14 times, and argued it was a misfire 7 times.
What I take from it
A "proof of done" gate is a good honesty filter and a weak quality filter. My agent's reports became trustworthy. The code did not get better. Making code better still takes tests that fail before the fix and checks that run against production after the deploy.
If I rebuilt the hook today I would check the transcript, not the message: accept quoted output only if a matching command actually ran in that turn, after the last edit. That closes the "paste output you never ran" hole and should cut the false positives on drafted text.
Measure your guardrails like features. Counting how often they fire tells you nothing about whether they work.
Caveats
- Stricter PR evidence rules shipped in the same batch as the hook, so the 21% to 6% drop is not the hook alone.
- "Self-certified" is my reconstruction (a fix PR whose red-run section says the failure was not reproducible). The original audit's figure had no saved method.
- The 60 labels come from one model pass with a few hand checks.
- A "fix" PR can repair an old bug, so the 1,072 number is a rework signal, not a defect count.
I help founders and small teams set up AI agent workflows and measure whether they work. If your agents say "done" more often than things are, DM me on LinkedIn or X.
Top comments (0)