Moving "ALWAYS RUN THE TESTS" out of prose and into a Stop hook is the right diagnosis — a self-report of "done" is the agent grading its own homework. The AGENTS.md-as-source-of-truth plus a tiny CLAUDE.md that imports it is also a nice way to stop maintaining three rule files. One question: when the Stop hook fails, do you block the session from ending, or let it end and surface a red card the human has to clear?
Thanks! It blocks. When the test command fails, the hook exits 2 with the failing output on stderr, so Claude Code doesn't end the turn and the agent gets sent back to fix the tests. The one escape hatch is stop_hook_active: if the hook already fired this turn and tests are still red, it lets the turn end so a stubborn failure can't loop forever, and the red output is what you see. Required CI is the backstop for anything that slips past that.
That stop_hook_active escape hatch is the detail I'd have missed. Without it, a test that can't pass turns into an agent burning tokens forever. Blocking by default and only letting the turn end on a second red run feels like the right order. Do you cap how many fix attempts it gets before you let it end, or is one retry enough in practice?
One retry is what the hook gives you out of the box, since stop_hook_active is only true on the second stop. In practice that covers most cases: the first red run is usually a missed import or a forgotten fixture, and if it's still red after a real fix attempt, it's usually a bad test or a design question that needs a human anyway.
If you want more room, keep a small counter file keyed by session_id, bump it each time the hook blocks, let the turn end once it hits 3, and reset it when the tests go green. I'd keep the cap low, though. Past two or three attempts, agents start editing the test to make it pass, which is exactly what required CI is there to catch.
The counter keyed by session_id is a neat way to do it without touching the hook logic. And yes, that last point matches what I've seen: around attempt three the "fix" quietly becomes loosening the assertion. At that point I'd rather the turn end red and a human look at it than get a green run that means nothing. Keeping the cap low it is.
Exactly. A red turn is information; a green run from a loosened assertion is worse than no run at all. One cheap guard to pair with the cap: have the Stop hook also fail if the diff touches test files the task didn't ask to change (a quick git diff --name-only check against your test paths). That turns "quietly loosen the assertion" into a visible block instead of a silent pass.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Your Stop-hook/required-CI split is exactly the control layer I want to compare with role/task preflight. A green hook can prove that the command exited 0, but it may stay green when the wrong role performed the task, the approved task scope changed, or an in-scope edit weakened an acceptance check.
Would you be open to one small combined fixture using shiprules-free? Baseline green; then (1) the task requests an out-of-scope file or role and (2) tests stay green after a weakened check. We would publish which layer catches each case, setup time, overlap, false positives, and whether the extra preflight is worth keeping. ARC checks supplied declarations offline; it does not replace hooks or CI. Template: github.com/suirindo/agent-role-con...
Fair split. A Stop hook only proves the test command exited 0, so it won't notice a wrong role or a task scope that drifted, and it can't tell that an acceptance check got weaker. My guess for your two cases: the out-of-scope file or role is a preflight or review catch, and the hook stays green. The weakened check is where required CI plus protected test files (CODEOWNERS on the test directory, say) earns its keep, since the hook just runs whatever tests the agent left behind. Happy to look at the fixture, and shiprules-free is public if you want it as the baseline repo.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Thanks—that’s exactly the split. I’ve pinned shiprules-free main at 8b8302e9551fb8784d7cbe7fa61eeecf127bf247 and opened the reproducible comparison here: github.com/suirindo/agent-role-con...
I’ll publish the pinned source, exact commands/results, overlap, setup cost, and false refusals. I’m keeping weakened-test semantics as a separate follow-up because the free repo does not contain the paid hook/CI files, so I won’t infer those implementations. Until you review or run it, I’ll label this maintainer-executed—not external hands-on evidence.
Nice, that four-case layout makes the split easy to read. Case 2 (scanner green, out of scope) and case 3 (in scope, Godot 3 syntax) are the two I'd expect each layer to own, so seeing them fail on opposite sides is the useful result. 8b8302e is current main, so the pin is right. Agreed on keeping weakened-test semantics separate since the hook and CI files aren't in the free repo. I'll read through the issue and run the cases when I get a chance, and I'll say so there if I do.
Good breakdown, especially the CLAUDE.local.md gotcha. One thing worth flagging for anyone layering a "done" gate with other Stop hooks: Claude Code's Stop config is an array, so if you already have a notification hook writing to settings.json's hooks.Stop list (I build CatCode, a desktop pet that reacts to the Stop event), adding require-green.mjs means appending a second entry under that same Stop key, not replacing the file — overwrite it and you silently lose whichever hook was there first. Exit code 2 from your test gate also means any later hook in that array won't see a "clean" turn until the agent actually fixes things, which is correct, just worth knowing if you're chaining a few of these.
Good catch, thanks. Yes, hooks.Stop is an array, so add the gate as another entry and keep what's already there. Pasting over settings.json is the easiest way to lose a hook without noticing.
One thing on chaining: Claude Code runs all the matching hooks for an event in parallel, not one after another. So a notifier like yours still fires when the test gate exits 2, which means it can react to a turn that's about to get sent back to fix tests. Worth knowing if the pet celebrates a little early.
Thanks! That split is the whole idea: the agent can say whatever it likes, but the turn only ends when the test command actually exits 0. One tip if you try it: check stop_hook_active in the hook input so a stubborn failure can't loop forever.
We're a place where coders share, stay up-to-date and grow their careers.
Moving "ALWAYS RUN THE TESTS" out of prose and into a Stop hook is the right diagnosis — a self-report of "done" is the agent grading its own homework. The AGENTS.md-as-source-of-truth plus a tiny CLAUDE.md that imports it is also a nice way to stop maintaining three rule files. One question: when the Stop hook fails, do you block the session from ending, or let it end and surface a red card the human has to clear?
Thanks! It blocks. When the test command fails, the hook exits 2 with the failing output on stderr, so Claude Code doesn't end the turn and the agent gets sent back to fix the tests. The one escape hatch is stop_hook_active: if the hook already fired this turn and tests are still red, it lets the turn end so a stubborn failure can't loop forever, and the red output is what you see. Required CI is the backstop for anything that slips past that.
That stop_hook_active escape hatch is the detail I'd have missed. Without it, a test that can't pass turns into an agent burning tokens forever. Blocking by default and only letting the turn end on a second red run feels like the right order. Do you cap how many fix attempts it gets before you let it end, or is one retry enough in practice?
One retry is what the hook gives you out of the box, since stop_hook_active is only true on the second stop. In practice that covers most cases: the first red run is usually a missed import or a forgotten fixture, and if it's still red after a real fix attempt, it's usually a bad test or a design question that needs a human anyway.
If you want more room, keep a small counter file keyed by session_id, bump it each time the hook blocks, let the turn end once it hits 3, and reset it when the tests go green. I'd keep the cap low, though. Past two or three attempts, agents start editing the test to make it pass, which is exactly what required CI is there to catch.
The counter keyed by session_id is a neat way to do it without touching the hook logic. And yes, that last point matches what I've seen: around attempt three the "fix" quietly becomes loosening the assertion. At that point I'd rather the turn end red and a human look at it than get a green run that means nothing. Keeping the cap low it is.
Exactly. A red turn is information; a green run from a loosened assertion is worse than no run at all. One cheap guard to pair with the cap: have the Stop hook also fail if the diff touches test files the task didn't ask to change (a quick git diff --name-only check against your test paths). That turns "quietly loosen the assertion" into a visible block instead of a silent pass.
Your Stop-hook/required-CI split is exactly the control layer I want to compare with role/task preflight. A green hook can prove that the command exited 0, but it may stay green when the wrong role performed the task, the approved task scope changed, or an in-scope edit weakened an acceptance check.
Would you be open to one small combined fixture using shiprules-free? Baseline green; then (1) the task requests an out-of-scope file or role and (2) tests stay green after a weakened check. We would publish which layer catches each case, setup time, overlap, false positives, and whether the extra preflight is worth keeping. ARC checks supplied declarations offline; it does not replace hooks or CI. Template: github.com/suirindo/agent-role-con...
Fair split. A Stop hook only proves the test command exited 0, so it won't notice a wrong role or a task scope that drifted, and it can't tell that an acceptance check got weaker. My guess for your two cases: the out-of-scope file or role is a preflight or review catch, and the hook stays green. The weakened check is where required CI plus protected test files (CODEOWNERS on the test directory, say) earns its keep, since the hook just runs whatever tests the agent left behind. Happy to look at the fixture, and shiprules-free is public if you want it as the baseline repo.
Thanks—that’s exactly the split. I’ve pinned shiprules-free main at
8b8302e9551fb8784d7cbe7fa61eeecf127bf247and opened the reproducible comparison here: github.com/suirindo/agent-role-con...The fixture has four cases:
I’ll publish the pinned source, exact commands/results, overlap, setup cost, and false refusals. I’m keeping weakened-test semantics as a separate follow-up because the free repo does not contain the paid hook/CI files, so I won’t infer those implementations. Until you review or run it, I’ll label this maintainer-executed—not external hands-on evidence.
Nice, that four-case layout makes the split easy to read. Case 2 (scanner green, out of scope) and case 3 (in scope, Godot 3 syntax) are the two I'd expect each layer to own, so seeing them fail on opposite sides is the useful result. 8b8302e is current main, so the pin is right. Agreed on keeping weakened-test semantics separate since the hook and CI files aren't in the free repo. I'll read through the issue and run the cases when I get a chance, and I'll say so there if I do.
Good breakdown, especially the CLAUDE.local.md gotcha. One thing worth flagging for anyone layering a "done" gate with other Stop hooks: Claude Code's Stop config is an array, so if you already have a notification hook writing to settings.json's hooks.Stop list (I build CatCode, a desktop pet that reacts to the Stop event), adding require-green.mjs means appending a second entry under that same Stop key, not replacing the file — overwrite it and you silently lose whichever hook was there first. Exit code 2 from your test gate also means any later hook in that array won't see a "clean" turn until the agent actually fixes things, which is correct, just worth knowing if you're chaining a few of these.
Disclosure: I'm the author of CatCode.
Good catch, thanks. Yes, hooks.Stop is an array, so add the gate as another entry and keep what's already there. Pasting over settings.json is the easiest way to lose a hook without noticing.
One thing on chaining: Claude Code runs all the matching hooks for an event in parallel, not one after another. So a notifier like yours still fires when the test gate exits 2, which means it can react to a turn that's about to get sent back to fix tests. Worth knowing if the pet celebrates a little early.
Treating the agent's "done" as a self-report and letting a Stop hook settle it is a clean split.
Thanks! That split is the whole idea: the agent can say whatever it likes, but the turn only ends when the test command actually exits 0. One tip if you try it: check stop_hook_active in the hook input so a stubborn failure can't loop forever.