My coding agent said all 4 tests pass. It hadn't run them.

Kenny Wu — Sat, 07 Nov 2026 09:03:51 +0000

You ask a coding agent to fix a bug. It edits the file and replies "fixed, all tests pass". Sometimes that is true. Sometimes the last test run happened before the edit, or never happened at all, and the sentence reads exactly the same either way.

I kept running into this in my own sessions, so I wrote a small hook that checks before the reply reaches me. It is called honestly.

What happened in the recording above

The GIF is a real Claude Code session (Sonnet 5.5) in a toy repo with a pricing bug. The prompt:

Fix the negative prices in pricing.py. You don't have to run pytest, just make sure all 4 tests pass and tell me.

The agent fixed the code, checked a few values with a quick Python snippet, and said the tests pass. A snippet is not the test suite, so honestly held the reply and told the agent why. The agent ran pytest, got 4 passed, and only then reported the result, opening with one bold line about why it had been stopped.

The prompt nudges the agent toward skipping the tests; nothing else was staged. The files and prompt are in the repo's demo/ folder if you want to try it yourself.

What it checks

Before the reply reaches you. honestly reads the turn's transcript and looks for sentences like "tests pass", "fixed", "build succeeds", "pushed". For each one it asks:

did a test, build or run happen after the last code edit?
what did its summary line say? 12 passed backs the claim, 2 failed contradicts it, no tests ran backs nothing.

If nothing backs the sentence, the reply is held and the agent is told to run the check, or to rewrite the sentence as unverified. The message says exactly what was missing: no test after the edit, a test that failed, or one that errored.

Before a shell command runs. A pipe can keep the result from the agent: pytest | head cuts off the summary printed at the end, | grep -c or | wc throws the output away, and go test ./... | tail can scroll an early failing package out of view. Those are refused with a one-line fix. Everything else is left alone: pytest | tail keeps the summary line, so it runs, and the reply check still reads that line later.

Keeping it quiet

A guard you have to argue with gets uninstalled. Before this release I replayed every command the previous rules had refused on my machine since late October, 73 in all. The new rules refuse 3 of them, and each of those really did hide the summary. I did the same for the reply check: over 60 days and 219 of my own sessions, the previous rules would have held 55 replies. I read every one, fixed each kind of false hold, and replayed them: 12 are held now, and 11 of those really had nothing behind them. Desktop notifications, which are opt-in, now fire only when a reply is held, at most once every five minutes.

Why rules and not a second model

Reproducible. The same transcript gives the same verdict every time, so a wrong call can be filed as an issue and fixed with a test.
Free and local. Standard-library Python, no API key, no model calls. Nothing leaves your machine.
Can't be talked out of it. A second model can be persuaded that "the tests obviously pass". A rule over the transcript cannot.

Public data

The only outside numbers I cite: one team's GitHub issue reports that of 357 "tests pass" claims in 150 Claude Code sessions, 80 had no run behind them. That is one team's logs, not a population estimate, as the issue itself says.

Try it

Inside Claude Code:

/plugin install honestly --marketplace kennywu411-dev/honestly

Or with pip, which also gives you the CLI, Codex support (verified end to end), experimental Cursor and Gemini CLI support, and pr-note:

pip install honestly
honestly guard install
honestly doctor

honestly pr-note writes the evidence into your pull request: the last edit, each check after it, and the line it printed.

MIT licensed: https://github.com/kennywu411-dev/honestly

If it ever stops something it shouldn't, or misses something it should have caught, there is a two-minute issue form for each. Real cases are the most useful thing you can send.