TL;DR — A loop is only as trustworthy as its stop condition. "Done" has to be checkable by a different model — tests pass, lint clean, schema validates — never the worker's own claim. Below: why self-grading fails, the four rungs of verification, a checker-agent template you can copy, and the three caps that stop a loop running forever.
Part 3 of the Loop Engineering series on ShipWithAI. Read the full article →
Part 2 gave you the parts list. Part 1 promised this installment would be entirely about designing stop conditions, "because they are the hardest part to get right." Here it is.
The thesis, in one line:
A loop that grades its own work will tell you it is done before it is.
Why self-grading fails
A stop condition is a claim about reality. If the agent that wrote the code is the only thing asserting the claim, you don't have a verification — you have a self-report.
The author's framing: a maker grading its own output is structurally optimistic, so it declares victory early. And this has a name — Anthropic's own writeup on building effective agents lists the failure mode explicitly: premature victory and fake-done features.
The four rungs
The rungs are an escalation of who checks:
| Rung | Mechanism | What it adds over the rung below |
|---|---|---|
| 1 | Bare while ! npm test; do claude -p ...; done + iteration cap |
A machine-checkable exit code as the gate |
| 2 | Stop hook (exit 2 or {decision:"block"}) |
Blocks completion until the check passes |
| 3 |
/goal — a separate model grades each turn |
A grader that is not the maker |
| 4 | Scheduled goal runs (Routines) | The condition is re-checked unattended |
Rung 1 checks with an exit code, rung 3 checks with a second model, and the gap between them is exactly the maker's optimism.
Worth knowing the mechanics of rung 2 vs 3: a Stop hook blocks completion with exit 2 until a check passes, and Claude Code auto-overrides after 8 consecutive blocks. /goal (v2.1.139+) instead has a separate model — Haiku — grade the stop condition each turn.
Rewriting a wish into a condition
"Run until the feature is done" is not a stop condition, it is a wish. A stop condition is a command with an exit code.
| Bad condition | Why it fails | Verifiable rewrite |
|---|---|---|
| "the feature is done" | Not falsifiable, no check | "all tests in test/auth pass AND lint clean" |
| "code looks good" | Not verifiable by a machine | "passes: true for every story in prd.json" |
| "the bug is fixed" | No boundary | "the new regression test exits 0 and no other test breaks" |
The three-attribute test for any condition you write:
- Verifiable — something can confirm it (a command, a count, a second model)
- Falsifiable — it can fail, with a clear failing signal
- Cheap — you can evaluate it every single turn
That last one is the one people underrate — "cheap" here means literally you can run it every turn, not that it's nice to have.
The checker agent
Copy this. The whole design is in what it's forbidden from doing:
---
name: stop-condition-checker
description: Grades whether the loop's stop condition holds. Not the maker.
model: haiku
---
You verify, you do not fix. Run the project's check command and report only:
- PASS if `bash tests/run.sh` exits 0 AND `bash scripts/lint.sh` exits 0
- FAIL otherwise, with the first failing line
Never edit code. Never report PASS on the maker's say-so; run the command.
The checker's value comes entirely from not being the maker. Different instructions, a cheaper model, and a clean context each turn are what make its PASS mean something.
The article's actual prescription: a checker with different instructions, optionally a different model, and ideally a clean context each turn. The different model is an option, not the requirement — the requirement is not being the maker. Splitting maker from checker is called "the single highest-leverage move in loop design."
One real trade-off worth knowing: the official ralph-wiggum plugin runs the checker in the same session as the maker, and the community has flagged that this deviates from fresh-context Ralph, where each lap starts clean. The article doesn't pick a side — same-session is cheaper; fresh-context is harder to fool. Pick deliberately.
The hands-on run
Read the disclaimer first, because it's the most useful part. The run did not use rung 3. /goal was the natural fit, but it's an interactive in-session grader and the author wanted a captured, reproducible run — so he used the rung-1 bare-loop equivalent: a fresh-context claude -p as the maker each turn, and the bash test exit code as the checker. An honest maker/checker split, graded by a machine-checkable artifact rather than by Haiku. He also notes token and dollar cost weren't instrumented, so he won't quote a number he doesn't have.
The bug, in ShipWithAI's content-agent repo: scripts/check-draft-seo.sh compares every keyword word against only the title's first three words — so any keyword longer than three words gets flagged, even when it leads the title verbatim.
The failing test:
⚠ keyword_placement: keyword "how to build an agent loop": not in title first 3 words
FAIL: multi-word keyword that leads the title was wrongly flagged.
exit=1
The stop condition — note that it guards against collateral damage, not just the target bug:
bash tests/test-check-draft-seo-multiword-keyword.sh exits 0
AND bash scripts/test-rubric.sh still exits 0
The fix, one line of real logic:
all_title_words = re.findall(r"[A-Za-z...]+", title.lower())
kw_words = re.findall(r"[A-Za-z...]+", kw_lower)
title_words = all_title_words[:max(3, len(kw_words))]
Result: one fresh-context maker turn. Pass on turn 1, cap of 8 never reached. He also ran a control draft to prove the fix isn't just a check that now always passes — worth copying as a habit.
The loop stopped because a command exited 0, not because the agent felt finished. That is the only kind of "done" you can leave a loop alone with.
Cap it three ways
The stop condition says when to stop on success. The cap says when to stop anyway. A loop you trust has both.
-
Max iteration count —
claude -p --max-turns N, plus the Stop hook's 8-block auto-override -
No-progress detection — continuous-claude's
--stall-threshold -
Budget ceiling —
--max-costand--max-duration
For lived precedent: ShipWithAI's own content-agent review loop caps at 3. The hands-on run above capped at 8. Pick per job, not by rule.
Before you try this
- A working harness and a minimal loop you can already run
- Claude Code v2.1.139+ for
/goal - A repo with a test command that exits non-zero on failure
Try it this week
Take the loop you built after Part 2 and ask one question: who asserts that it's done? If the answer is "the same agent that did the work," you haven't reached rung 1 yet.
Write the condition as a shell command. Add a second agent whose only job is to run it. Then pick your three caps.
This is a condensed summary. The full article walks each rung with working config, the complete hands-on trace, and the FAQ on /goal vs Stop hooks:
👉 Stop Conditions: Making "Done" Mean Something — Part 3, ShipWithAI
Earlier: Part 1 — Why You Should Stop Prompting · Part 2 — Anatomy of a Loop. Part 4 — Memory Outside the Context Window — ships next.
Top comments (0)