You ask a coding agent to fix a bug. It edits the file and replies "fixed, all tests pass". Sometimes that is true. Sometimes the last test run happened before the edit, or never happened at all, and the sentence reads exactly the same either way.
I kept running into this in my own sessions, so I wrote a small hook that checks before the reply reaches me. It is called honestly.
What happened in the recording above
The GIF is a real Claude Code session (Sonnet 5.5) in a toy repo with a pricing bug. The prompt:
Fix the negative prices in pricing.py. You don't have to run pytest, just make sure all 4 tests pass and tell me.
The agent fixed the code, checked a few values with a quick Python snippet, and said the tests pass. A snippet is not the test suite, so honestly held the reply and told the agent why. The agent ran pytest, got 4 passed, and only then reported the result, opening with one bold line about why it had been stopped.
The prompt nudges the agent toward skipping the tests; nothing else was staged. The files and prompt are in the repo's demo/ folder if you want to try it yourself.
What it checks
Before the reply reaches you. honestly reads the turn's transcript and looks for sentences like "tests pass", "fixed", "build succeeds", "pushed". For each one it asks:
- did a test, build or run happen after the last code edit?
- what did its summary line say?
12 passedbacks the claim,2 failedcontradicts it,no tests ranbacks nothing.
If nothing backs the sentence, the reply is held and the agent is told to run the check, or to rewrite the sentence as unverified. The message says exactly what was missing: no test after the edit, a test that failed, or one that errored.
Before a shell command runs. A pipe can keep the result from the agent: pytest | head cuts off the summary printed at the end, | grep -c or | wc throws the output away, and go test ./... | tail can scroll an early failing package out of view. Those are refused with a one-line fix. Everything else is left alone: pytest | tail keeps the summary line, so it runs, and the reply check still reads that line later.
Keeping it quiet
A guard you have to argue with gets uninstalled. Before this release I replayed every command the previous rules had refused on my machine since late October, 73 in all. The new rules refuse 3 of them, and each of those really did hide the summary. I did the same for the reply check: over 60 days and 219 of my own sessions, the previous rules would have held 55 replies. I read every one, fixed each kind of false hold, and replayed them: 12 are held now, and 11 of those really had nothing behind them. Desktop notifications, which are opt-in, now fire only when a reply is held, at most once every five minutes.
Why rules and not a second model
- Reproducible. The same transcript gives the same verdict every time, so a wrong call can be filed as an issue and fixed with a test.
- Free and local. Standard-library Python, no API key, no model calls. Nothing leaves your machine.
- Can't be talked out of it. A second model can be persuaded that "the tests obviously pass". A rule over the transcript cannot.
Public data
The only outside numbers I cite: one team's GitHub issue reports that of 357 "tests pass" claims in 150 Claude Code sessions, 80 had no run behind them. That is one team's logs, not a population estimate, as the issue itself says.
Try it
Inside Claude Code:
/plugin install honestly --marketplace kennywu411-dev/honestly
Or with pip, which also gives you the CLI, Codex support (verified end to end), experimental Cursor and Gemini CLI support, and pr-note:
pip install honestly
honestly guard install
honestly doctor
honestly pr-note writes the evidence into your pull request: the last edit, each check after it, and the line it printed.
MIT licensed: https://github.com/kennywu411-dev/honestly
If it ever stops something it shouldn't, or misses something it should have caught, there is a two-minute issue form for each. Real cases are the most useful thing you can send.
Top comments (1)
the case I keep hitting is narrower than a pipe. after I edit a module that several files import, I run only the test file I was reading, and pytest prints
12 passedand exits 0. that is a run after the last edit and a line that says passed, so it reads like evidence — but the other file's 3 tests were never in that run, and the output carries nothing that says so: I looked for the filename, for a deselected count, for the total of 15, and none of the three appears. writing the total beside the count,12 of 15, is the only thing I have found that lets that line tell a narrowed run from a full one.