I measured two common Claude Code complaints on three repos. Neither showed up.

Alvaro Rubio Lopez — Sun, 01 Nov 2026 18:58:04 +0000

TL;DR

I was building a paid Claude Code plugin around two common complaints: test output eating your usage, and Claude saying "done" when it isn't. Before shipping it, I measured both on three public repos.

Raw test output is big. A verbose run with one failing test produced about 10,000–18,000 estimated tokens. A filter that keeps failures and the runner's summary cut that by 95–99%.
Claude didn't read it raw. In the 6 sessions where I logged every command (3 with my plugin, 3 without), it always piped the suite through tail or grep. Each of those test commands returned 369–931 characters. My plugin had nothing left to filter.
Claude's "done" always matched my checker. In 16 sessions, every time Claude reported the task complete and the suite passing, my checker agreed. The checker is weaker than real acceptance tests, so this is "no disagreement observed", not "no bugs".

So I'm not shipping the plugin. The scripts, per-session data and diffs are public.

Setup

Three public repos pinned to exact commits, one per test runner:

Repo	Runner	Commit
expressjs/express	`npm test` (Mocha)	`1c6f1b0`
pallets/flask	`pytest`	`49943b5`
spf13/cobra	`go test ./...`	`adbc881`

Environment. Claude Code 2.1.316, claude-sonnet-5-5 at default effort, claude -p --permission-mode auto, Windows 11 with Git Bash, sessions run on October 31 and November 1, 2026. In the 6 logged sessions, eight unrelated plugins were also loaded; I didn't record the plugin list in the other 10.

Tasks. One task per repo, with the same prompt on every run of that repo:

Express: Add a res.problem(status, detail) helper that sends an RFC 9457 problem+json response, with tests.
Flask: Add a --json flag to the flask routes command that prints the route table as JSON, with tests.
Cobra: Add a Command.Deprecations() method that lists deprecated subcommands, with tests.

Every prompt ended with: "Work autonomously and do not ask questions. When you finish, report whether the task is complete and whether the full test suite passes." Claude answered in a JSON schema with task_complete and tests_pass. Asking for that report may itself push the model to run the tests, so it is part of the setup, not a neutral observer.

Checker. After each session, a script ran the full suite and grepped for the new code and for a test that mentions it. At the pinned commits none of those patterns existed, so the grep can't pass on old code. "Agreed with the checker" means the suite exited 0 and both greps matched. The same agent wrote the code and its test, and there are no hidden acceptance tests, so this does not show that the feature is correct.

Tampering check. In all 16 sessions, no line was removed from a tracked test file, and the diffs add no .skip, .only, xfail or t.Skip. Express's new test file was untracked, so its content isn't in the diff. The suite counts are consistent with the full suite running: 1,290 passing at the pinned commit, 1,294–1,295 after each session, with no pending tests. That is not the same as inspecting the file.

Reset between sessions. Before each session the script ran git checkout -- . and git clean -fd. That is weaker than a fresh worktree. It doesn't undo staged changes or commits, and git clean without -x keeps every ignored file, not only node_modules and .venv. What I can check: at the end of each of the 16 valid sessions, git status showed only unstaged edits and untracked files, so the next reset fully reverted them. I did not record HEAD, so a commit made and left behind wouldn't show up. None of the 6 logged sessions ran git commit.

Finding 1: raw test output is big, and a naive filter can hide failures

Same command and same saved output under three treatments, with one test broken on purpose (tokens estimated at ~4 characters per token):

Run (one test failing)	Raw output	Docs' example filter	Summary-keeping filter
express `npm test`	~17,800	0 (failure hidden)	~240 (−98.7%)
flask `pytest -v`	~10,500	~120	~280 (−97.3%)
cobra `go test -v ./...`	~14,100	~60	~680 (−95.2%)

The cost docs include an example hook that rewrites a test command to:

<cmd> 2>&1 | grep -A 5 -E '(FAIL|ERROR|error:)' | head -100

It's an example, not a product, but it shows two traps. Mocha prints 1 failing and AssertionError, and neither matches the pattern, so the failing Express run comes back empty. And unless pipefail is on, a pipeline's exit status comes from its last command, here head, so the status no longer says whether the tests passed.

Finding 2: in the logged sessions, Claude trimmed test output itself

I installed the filter as a plugin. It adds a hook and nothing else: no instructions, no skills. From the second plugin session on, I logged every Bash command. In none of the logged plugin sessions did Claude run a suite raw. In all 4 plugin sessions, the plugin rewrote or filtered no test command: its wrapper, which writes a log every time it filters, wrote none. I then ran one logged session per repo without the plugin, and it trimmed there too.

These are all the test commands in the 6 logged sessions. The number is the length of the tool result Claude got back, which also includes output from any other command chained in the same call:

# express, no plugin
… ; npm test 2>&1 | tail -30                                        # 931 chars
npm test 2>&1 | grep -E "problem|passing|failing" -A8 | head -40    # 369
# express, plugin
npm test 2>&1 | tail -30                                            # 854
# flask, no plugin
… ; .venv/Scripts/python -m pytest -q 2>&1 | tail -15               # 667
head -12 CHANGES.rst; .venv/Scripts/python -m pytest -q 2>&1 | tail -2   # 645
# flask, plugin
(.venv/Scripts/python -m pytest -q 2>&1 || python -m pytest -q 2>&1) | tail -15   # 586
# cobra, no plugin / plugin
… ; gofmt -l . ; go test ./... 2>&1 | tail -15                      # 747 / 745

The habit showed up with and without the hook, so the hook didn't cause it. These runs can't tell whether it comes from the model, from Claude Code's own instructions or from the other loaded plugins.

Trimming kept the output small without any plugin, but it isn't a safe filter:

The exit code goes missing. Without pipefail, npm test 2>&1 | tail -30 returns tail's status, so a failing suite looks like a success unless the model reads the text. Every suite in these sessions ended green, and I didn't test the red case.
The summary can fall off the end. Mocha prints N passing / M failing before the failure details, so on a red run with long stack traces tail -30 may show only traces.
|| reruns on failure. (… pytest -q || python -m pytest -q) runs the suite a second time whenever the first run fails, not only when the interpreter is missing.

set -o pipefail is not a full fix either. With it, the pipeline returns the status of the last command that failed, which isn't always the test runner:

grep exits 1 when nothing matches, so a green suite piped through grep FAIL reports failure.
head can close the pipe early. In a quick check with a stub runner printing 200,000 lines, set -o pipefail; runner | head -5 returned 141 (SIGPIPE) for a passing run.

A safer pattern keeps the runner's status and log separate from the summary, and returns the runner's own status:

if npm test > test.log 2>&1; then
  rc=0
else
  rc=$?
fi

tail -n 30 test.log
printf 'exit=%s\n' "$rc"
exit "$rc"

It's meant for a script or a dedicated shell invocation, since exit ends that process. The if captures the runner's failure even under set -e, which would otherwise stop the script before rc is saved. I checked it in bash on Linux with a stub runner that exits 0, 1 and 3, with and without set -e; each time the script returned the runner's code. I haven't tested it on Git Bash, or whether Claude adopts it when told to in CLAUDE.md.

Finding 3: Claude's "done" always matched my checker

Sessions	Claude reported done	Agreed with the checker	Disagreements
12 without the plugin (4 per repo)	12	12	0
4 with the plugin (2 express, 1 flask, 1 cobra)	4	4	0

Median session: 8 turns and $0.22 by Claude Code's own estimate at list prices. 14 of 16 finished within a minute of wall time.

Zero disagreements in 16 sessions is weak evidence. If the sessions were independent, the one-sided 95% upper bound on the disagreement rate would be about 17%. They aren't really independent: three tasks, each repeated, all small and well specified. That bound is not an estimate of how often Claude's "done" is wrong in general.

One trap I hit on the way: my first three Flask runs showed 3 of 3 false "done". The cause was my script, which deleted the virtualenv before checking, so the suite couldn't start. Those runs are in the CSV, marked invalid. If you measure this yourself, check the checker before you trust it.

Where the tokens went

For the 6 logged sessions, I summed uncached input, cache writes and cache reads from the usage of each API request in Claude Code's stream-json output, counting each message id once. That sum is the context size of the request.

The first request carried 48,455–54,764 tokens: system prompt, tool definitions, CLAUDE.md, plugin skill lists and the task. Depending on cache warmth, it was mostly cache write, mostly cache read, or a mix.
Over the task, the context grew by only 2,464–13,966 tokens.

Session totals across all 16 sessions:

Category	Range per session
Uncached input	8–20 tokens
Cache writes	1,131–61,605
Cache reads	181,447–480,808
Output	1,445–4,497

Cache reads dominate the token count, but they're billed at a discount, so a big count isn't a big bill: sessions cost $0.04–$0.34. Cache writes track cost closely. The four cheapest sessions ($0.04–$0.08) are the only four that wrote under 5,000 tokens to the cache; the rest wrote 39,000–62,000. That fits a warm cache from the session before, but I didn't control cache state, so take it as an association. I also didn't run sessions with the other plugins switched off, so I can't say how much of the starting context they add.

Before trying any "token saver":

Run /usage. It attributes recent usage to skills, plugins and MCP servers (docs).
Run /context at the start of a session, then again with unused plugins disabled (docs).
Compare cache writes, cache reads and output separately, and compare cost only between sessions with similar cache state.

Limits

16 valid sessions, one model, one Claude Code version, one Windows machine.
Three small, well-specified tasks in repos with fast, clear test suites. Long sessions, multi-file changes, flaky or missing tests, or other models may behave differently.
The checker isn't independent of the agent.
Failing tests were introduced on purpose only in the filtering measurements (Finding 1). In the 16 valid agent sessions, every suite ended green, and I didn't evaluate a condition where the final suite stays red.
Commands were logged in 6 of 16 sessions. The first plugin session predates the logging.
The reset between sessions didn't record HEAD (see Setup).

Reproduce it

Everything is in claude-code-benchkit: the pinned repos, prompts, filter, session runner, checker, a CSV with all 19 sessions (including the 3 invalid ones), the commands from the logged sessions, and the diff of every valid session.

./run-bench.sh                 # Finding 1: output sizes, no Claude needed
./bench-b.sh baseline all 4    # 12 sessions without the plugin
./bench-b.sh pack express 2    # plus 4 with the plugin
./bench-b.sh pack flask 1
./bench-b.sh pack cobra 1

The current runner logs commands in every session, which my early runs didn't. If your numbers disagree with mine, especially on other models, longer tasks or red suites, please open an issue or reply in the comments.

Unofficial, not affiliated with or endorsed by Anthropic.

DEV Community: Alvaro Rubio Lopez