Shopify's Helix post went around my feeds for two weeks with the same sentence attached: "An attempt is allowed to be wrong. It is not allowed to ship until it isn't." Four gates on every AI checkpoint, in order. Tests, a design review, two independent reviewer agents, an engineer.
I read the post twice looking for a number. There is none. No pass rate, no cost, no count of what escaped. Twelve weeks for the Shop app and more than 300 screens is the whole ledger.
So I built the gates in Claude Code on a repository with a real test suite, and asked the only question the post leaves open: which of the four gates actually catches bugs, and what does each one cost?
What I actually tested
The repository is tinydb, a small document database with 226 tests green on the commit I used. The worker is Sonnet 5.5 run headless with claude -p, at most 60 turns, edits auto-accepted.
Five requests, typed the way a developer types them: push and pull operations on a field, a between query, a table rename, a sorted search, a write cache size. Behind each request sit five hidden requirements the agent never sees, the things a careful senior engineer would expect. That makes 25 requirements, checked by hidden tests that live outside the repository.
Five configurations: no gate, acceptance tests alone, two reviewers alone, tests then reviewers, tests then a reviewer paired with a critic. Two repetitions per task per configuration. 50 runs. The design gate is skipped: this is a library, and Shopify's post says a logic-only change simply skips it.
Every gated configuration runs through the same Stop hook.
A Stop hook is a script that exits 2
Anthropic's own best-practices guide names the problem: without a check, "looks done" is the only signal available, and you become the verification loop. A Stop hook runs every time the agent tries to end its turn. It lives in the project settings:
{
"hooks": {
"Stop": [
{
"hooks": [
{
"type": "command",
"command": "bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"",
"timeout": 1500
}
]
}
]
}
}
The contract is the exit code. 0 lets the turn end. 2 blocks it, and whatever the script printed to standard error goes back to the agent as the reason. On screen: the agent sums up its change and stops, the hook runs the acceptance tests, one fails, the turn does not end. The agent reads the failure, fixes the code, stops again. This time the gate is quiet.
A rule in the prompt is advice. A hook is not. Yet the same documentation page lists three ways out.
cap 8 consecutive blocks, then Claude Code overrides the next one
(resets on every tool call; CLAUDE_CODE_STOP_HOOK_BLOCK_CAP raises it)
timeout 600 s by default; a hook that times out renders no decision
flag stop_hook_active is true when the session is already continuing
because of a stop hook; the docs say to check it
My gate logs the flag and ignores it, and gives up on its own after six blocks so the loops stay countable.
There is a fourth exit nobody documents. After being blocked once, the agent went looking for the tests: cat on the hook script, then find across the whole disk for the test file. Only the permission deny list stopped it. So the acceptance tests live outside the repository tree, the settings deny reads of the hook and of the hidden tests, and the agent sees the failure but never the file.
Acceptance tests took the pass rate from 70% to 84%
With no gate at all, 35 of the 50 requirements pass. Three runs out of ten are fully clean.
Shopify's first gate is not the agent's own tests. A subagent writes integration-style tests per checkpoint, and those define what proven means. So the writer here is a separate headless Sonnet that sees the request and the repository before the change, read-only, and cannot run code. It writes between 23 and 53 tests, 39 on average, in about 40 seconds.
Tests alone: 42 of 50, six fully clean runs instead of three.
task no tests with tests
sorted search 5 of 10 10 of 10
write cache size 8 of 10 10 of 10
push and pull 4 of 10 4 of 10
The sort task shows what the misses look like. The agent indexed every document with the sort field, and the moment one document lacks it the whole search crashes. The writer asks exactly that kind of question. The push and pull task does not move: two of its three misses are about a document without the field, the request never said anything about that, and the writer's test accepts a raise or a silent no-op. The gate cannot recover a requirement nobody stated.
Across the 30 gated runs with tests, the gate sent the agent back 28 times with a real requirement failing and five times with all of them passing. The price is $0.20 a run instead of $0.085, and 60 seconds instead of 33.
One detail I would not skip: my first version of this gate reran the implementer's own tests. It allowed every run on its first stop while a hidden requirement failed. An agent's own tests test what it built.
Two reviewers added nothing, at three times the cost
Shopify's wording: two independent, context-isolated reviewer agents, every finding fixed, the loop repeats until both approve. Each reviewer here is a headless Sonnet with read-only tools and at most 15 turns, returning structured JSON: an approve flag and findings with a severity of blocking or minor. Two run in parallel on the request and the full diff, their findings are combined, and the gate blocks when either one finds something blocking. A pair costs between 3 and 6 cents per stop.
configuration passed rate blocks
no gate 35 of 50 70% none
reviewers alone 34 of 50 68% 1 in 10 runs
tests alone 42 of 50 84% 6 in 10 (3 right, 3 false)
reviewers after tests 41 of 50 82% 2 in 10, both right
The sort task explains the number. In one run both reviewers blocked on the key error crash, the agent fixed it, five of five. In two other runs both reviewers saw the exact same crash and rated it minor, because the docstring documented it. Both approved, both runs ended at zero of five. Same model, same diff, same bug, two severities. The reviewers were never wrong when they blocked. The problem is what they wave through.
A paper from August measured the same shape on LiveCodeBench: zero-shot 77%, one reviewer 77%, two independent reviewers 75%. Reviewer plus critic, where a second agent audits the review before anything is edited, 87%. So I built that too. Reviewer A reviews, critic B stamps each finding upheld or rejected, only an upheld blocking finding closes the gate. On this repository: 39 of 50, 78%, and zero blocks in ten runs.
What no reviewer shape fixed is the loop. Three runs hit six blocks on the same acceptance test and the gate gave up with the requirement still failing. In the one worth remembering, the test wanted a failing update to leave every document untouched, so the agent rewrote the table's update method. The reviewers blocked that as an unrequested change. The agent reverted and wrote that the acceptance test and the code review were asking for opposite things, so it had stopped without changing anything.
What this doesn't prove
One repository, one worker model, five tasks, two repetitions per cell. A gap of two points between rows is noise. The fourteen points between no gate and the test gate is not. The design gate was not built and the engineer was not timed. The whole matrix cost $9.95 of API spend as the CLI reports it.
configuration passed cost/run left for the human
no gate 35 $0.085 15
acceptance tests 42 $0.20 8
reviewers alone 34 $0.15 16
tests with reviewers 41 $0.28 9
tests with reviewer + critic 39 $0.28 11
The full run
The video shows the hook firing on a real failing acceptance test, the agent hunting for the hidden file, and the two reviewers rating the same crash minor.
My verdict for a repository like this one: keep the test gate with a writer that is not the implementer, give the loop a counter, and keep reviewers as advice in the feedback rather than as a bar. If you run a Stop hook today, what does it block on? A) the agent's own tests B) tests someone else wrote C) a reviewer agent D) nothing yet, I am the gate.
An AI assistant helped tidy the prose. The runs, the numbers and the opinions are mine.
Top comments (0)