Discussion on: Your AI agent says "done". Your tests say 3 failing. The Stop hook that settles it

[asagiman]

Thanks—that’s exactly the split. I’ve pinned shiprules-free main at 8b8302e9551fb8784d7cbe7fa61eeecf127bf247 and opened the reproducible comparison here: github.com/suirindo/agent-role-con...

The fixture has four cases:

1. Clean Godot 4 input inside the declared fixture path → ARC PASS, scanner PASS.
2. Scanner-green edit outside the declared task/role scope → ARC expected FAIL, scanner expected PASS.
3. In-scope Godot 3 syntax regression → ARC scope expected PASS, scanner expected FAIL.
4. Repair → both PASS.

I’ll publish the pinned source, exact commands/results, overlap, setup cost, and false refusals. I’m keeping weakened-test semantics as a separate follow-up because the free repo does not contain the paid hook/CI files, so I won’t infer those implementations. Until you review or run it, I’ll label this maintainer-executed—not external hands-on evidence.

[ShipRules]

Nice, that four-case layout makes the split easy to read. Case 2 (scanner green, out of scope) and case 3 (in scope, Godot 3 syntax) are the two I'd expect each layer to own, so seeing them fail on opposite sides is the useful result. 8b8302e is current main, so the pin is right. Agreed on keeping weakened-test semantics separate since the hook and CI files aren't in the free repo. I'll read through the issue and run the cases when I get a chance, and I'll say so there if I do.