You ask Claude Code, Codex, Cursor, etc. for a quick fix. It changes the code and says "Fixed ✅". Later you find out it never ran a single test, or...
For further actions, you may consider blocking this person and/or reporting abuse
rejecting | tail as proof is the part that matters. that's the exact shortcut an agent would take.
Exactly! The agent isn't lying on purpose.
npm test | tailreally does print the last lines, but the exit code that reaches the shell is tail's, and that's always 0. So a failing test run looks green. proof-of-done checks for that: if the exit code is hidden by a pipe,|| trueor;, it blocks "done" and asks the agent to run the test command again on its own. The only exception isset -o pipefail, because then the real failure still comes through. Thanks for reading! 🙌pipefail is the exception that saves it. most wrappers never get there.
Right, and the plugin doesn't guess: a piped run only counts when
set -o pipefailis in that same command, so the real failure can still reach the shell.same command is the part people miss. run it separately and the failure still gets swallowed.
Yes, exactly! In Claude Code each Bash call starts a fresh shell, so
set -o pipefailin one command is gone by the timenpm test | tailruns in the next. That's why the plugin only accepts it inside the same command. Thanks for following this thread so closely! 🙌The 0-tests-ran rule is the one I'd want most. A green run with nothing behind it is worse than a red one — at least red is honest. Does the flaky flag ever fire on legitimately order-dependent tests, or does it stay quiet unless the same check flips with no edit in between?
Thanks, Paul! And agreed, a green run with nothing behind it is worse than a red one. To your question: it stays quiet unless the same check fails and then passes after your last edit, with no edit in between. It doesn't know about test order. One case where it does fire: a single file's tests fail and then the full suite passes. That's often an order-dependent test, so it gets flagged. It doesn't force a fix either: it only asks the report to say the check is flaky.
Blocking stop until something actually runs after the last edit is the bit I keep wishing Claude Code had by default. The SAVE10 fix that then blew FLAT50 into −₹20 is such a clean case for why "Fixed ✅" without proof is worse than silence.
Thank you! That SAVE10/FLAT50 bug is exactly why I built it: the fix looked right, the report said done, and only running the checks shows what broke.
The "0 tests ran → blocked" rule has a twin in data work, and I walked into it today. I fired off a batch of search runs, and 31 of them finished with status SUCCEEDED while each returned a single note row: the date window was too wide for a text search, so nothing had run. Anything reading the run status would have reported 31 empty results, which reads as "nobody asks for that skill". Same shape as a green suite with no tests in it.
What caught it was the question your plugin asks: not "did it exit 0" but "did the thing I care about happen". For a data pull that means "did any row come back that isn't a note". I now check that before I read a single number.
Love this example. "SUCCEEDED with nothing behind it" is exactly the same shape as a green suite with 0 tests. Checking "did a real row come back" before reading any number is the right instinct. Thanks for sharing it!
The reason string is doing real work in a block like this. In a test of ours on Claude Code 2.1.294, a PreToolUse hook that only logged and exited 0 let Sonnet 5.5 edit a protected JSON file. The same hook returning permissionDecision deny with a reason blocked the edit, and the model then proposed the migration the reason named instead of writing it.
Agreed, the reason is the useful part. That's why every block in proof-of-done says what to do next ("run this command again on its own", "say in your report that it's flaky") instead of a plain no.
rejecting zero test runs and catching
|| trueis the part most setups miss, nice. we started from the same idea, don’t take the agent’s word for “done”, but kept hitting the limit that a check like this is only as good as the tests that exist. nobody writes a test for a missing timeout until it pages someone. so we went the other way and had it read the whole codebase instead of the test results, looking for reliability risks directly. a payment call with no timeout, a retry on a write with no idempotency key, a connection that isn’t released on the error path. that became PRI (Production Reliability Index), a 0 to 100 score per change at Tomosu with a before and after fix for each finding. it runs through MCP in Claude Code too, if you wanted to take a look tomosu.ai/mcpThank you for the kind words, Parsa, and for sharing your experience. You raise an important point: a verification gate can only be as strong as the tests behind it. That's one reason proof-of-done 0.4 also asks whether a test should be added when code files change without any test change, so the gap is visible before the task is reported as done. If it's useful, here's a 36-second demo of how it works in a real session: youtube.com/shorts/ozlYRXQTs-U
the pipe check is the sneaky one.
| tailswallowing the exit code is how we lost an afternoon once, the run looked green while the failures sat upstream of the pipe.the zero tests ran case is just as common though, a bad glob or a renamed folder and jest happily exits 0. how does it behave in a monorepo where the changed package has tests but the root run only picks up a different workspace?
Can a passing lint run clear the gate after a test run fails? I'd want those tracked separately. Fixing a formatting issue shouldn't let the agent call the job done while the test failure is still unresolved.
You're right to want them tracked separately, and your comment made me close the last gap. As of proof-of-done 0.4.2, a check that failed has to run again after the last edit: a passing
npm run lintcan't clear a failingnpm test, even if the agent edits code in between. The stop stays blocked until the test runs again, or the report says honestly why it can't. To update:/plugin marketplace update claude-code-skills