The bill keeps climbing. Every span in the trace comes back 200 OK. The dashboard says the agent is healthy. Where is the money going?
I built a small CLI called Clew to answer that. What I found — over six rounds of measurement on public trace data — surprised me. Claude Code itself, session after session, is not the problem. The waste I was hunting is somewhere else. And the most promising detector I built failed hard enough that I had to kill it.
Why I built it
What I actually wanted to see was the black box between agents — the part where one hands off to another and you have no idea what got repeated, re-explained, or thrown away in the gap. But you can't measure what happens between two agents until you can measure what happens inside one. So I started there, with my own Claude Code sessions, and the detour turned out to be the interesting part. Every time I was sure I knew where the waste was, the data said otherwise.
How it decides — no LLM in the loop
Two gates, both deterministic:
- Structural gate — group spans by (node, normalized input); groups of >= 2 are candidates.
- Identity gate — require sha256(output_A) == sha256(output_B). If outputs differ, NOT flagged.
If the outputs are byte-identical, the pair is a candidate. If they differ — state changed, a retry succeeded where the first failed, the tool returned a slightly different string — the pair drops out. No embedding, no judge, no model in the loop.
On top of each flagged pair, a report-only category label chosen strictly from the tool name:
error_repeat — the response matches an error pattern (agent re-ran with the same wrong args).
side_effect — a state-changing tool (Edit, github-create_pull_request, …) re-invoked with the same arguments.
idempotent — a read-only or declarative tool (Read, filesystem-list_directory, …) called repeatedly.
unclassified — the tool's effect depends on the payload. Bash, PowerShell, local-python-execute, bigquery_run_query live here. The name alone cannot classify them; I don't try.
Labels never change the cascade. The sha256 gate is authoritative; the label is guidance for the reader.
Frozen parameters: phi = 0.514345, N = 2, embedding model pinned to a git tag with a manifest sha256. N = 2 was an arbitrary default at first; I later checked it against N in {2, 3, 5, inf} on RedundancyBench and it turned out F1-optimal (F1 decreases monotonically as N grows). That is the only tuning story in the repo, and it's post-hoc verification, not a knob I turned.
What I measured
RedundancyBench (human-labeled ground truth)
Step-level F1: 0.2642 vs the paper's best LLM-as-judge method at 0.2488. Precision 0.826. Same evaluate.py, same scope, imported directly from their repo.
Honest caveats: the 0.2488 comparison is the paper's cited number — the baseline's prediction files aren't in the repo, so I can't reproduce 24.88% locally. Recall is 0.157. Clew catches one narrow pattern well and deliberately ignores the rest.
trace-commons (28 real public Claude Code sessions)
Full scan, 2026-07-19:
28 / 28 sessions processed, 0 crashes.
10 / 28 flagged as wasteful. 18 / 28 had zero flagged waste.
Aggregate saving potential across all wasteful sessions: $1.01 to $10.12 (cache-hit to cache-miss).
That is a real, low number. The median CC session in this set has no detectable waste at all.
Toolathlon (6,780 trajectories, 22 frontier models × 3 runs)
Now the picture flips. Over 176,270 tool spans, Clew flags 8,042 duplicate pairs.
Self-rebuttal, up front: 47% of those (3,791 pairs) are idempotent — the grey area where "is this really waste?" depends on whether external state changed between the calls. Strip the grey area out and you get:
4,251 pairs (2.41% of tool spans) — about 3× the rate on CC sessions (0.80%).
1,343 side_effect pairs — state-changing tools re-invoked with matching arguments.
Including 459 duplicate email-send pairs. This is detection of duplicate invocations, not confirmation the SMTP server actually delivered twice — the trace can't tell me that. 459 same-argument email-send pairs, in a benchmark, from frontier models.
Toolathlon adapters carry no token counts, so no dollar figure for these. And Toolathlon ships only pass/fail labels, not step-level ground truth. Candidate density varies 54× across models (0.157 – 8.463 per trajectory), but "model X wastes 54× more than model Y" would over-claim — no labels, task-mix and success-rate confounds uncontrolled.
The twist
I built this expecting Claude Code — my daily driver — to be full of waste. So I went looking, one candidate pattern at a time, on real CC sessions:
Normalization gaps — same call, different JSON key order or path format, so the hashes miss each other. Effect: zero. The adapter already canonicalizes before hashing.
File re-reads — reading a file whose content is already in context. Effectively zero real cases. Pre-registered a detector for it anyway; it died. (Next section.)
Dumb retries — failing, then re-running with identical arguments. Almost absent. CC reads the error and changes the arguments.
Context accumulation — this one is real. A file read once stays in the trajectory and gets re-consumed on every subsequent turn, so a single read early in a long session is paid for many times over. But "was this context still needed 100 turns later?" is a judgment call, not a hash comparison. It's outside what a deterministic tool can claim.
Inefficient sessions having more duplication — I expected sessions that burned more tokens per unit of work to also duplicate more. No correlation.
Read↔Edit ping-pong — one file in one session was opened 76 times. Looked damning. It was normal incremental development; each edit touched a different part of the file.
Five of six weren't there. The sixth is real but sits outside the line I drew for myself — no LLM judge, ever.
Then I pointed the same tool at Toolathlon, where agents work with 523 distinct tools instead of CC's 20, and the rate tripled. The pattern: Claude Code by itself is efficient. Waste appears when you stack many tools that look interchangeable to the model. A codebase with ~20 tools rarely gets confused. A benchmark with 523 tools does. That's a claim about tool orchestration, not about any one model's intelligence.
The KILL story
The most interesting failure I've had on this project.
In real CC sessions, the agent sometimes re-reads a file it already read, with no Edit in between. Clear waste, right? I pre-registered a "file re-read" detector: two Read spans on the same canonical path, no writer tool between them, no Bash / PowerShell between them (which could modify the file behind our back).
Pre-registration matters here — the predictions and stop conditions get pushed and PR'd before the results run. K4 threshold: 70% precision on a 30-pair random sample. Written before I saw any output.
Reproduced 2026-07-24. Pool of 209 gated Read pairs. Random seed 42. 30 pairs sampled.
1 of 30 pairs had identical arguments.
The other 29 were the agent reading different slices of the same file — offset=163, limit=100, then offset=260, limit=40, and so on. Normal workflow. Not waste.
The 1 that matched was a password-protected PDF. Both outputs, byte-identical:
PDF is password-protected. Please provide an unprotected version.
That's a failed-tool retry pattern, not a file re-read.
Lenient precision (count the PDF): 3.3%
Strict precision (don't): 0%
Threshold: 70%
KILL. find_reread_candidates removed from src before v0.3.0 shipped. Branch retained for provenance, never merged.
Then the natural follow-up: what if I add an args_equal gate — require offset and limit to match exactly? Across the full 209-pair pool, that gate found 6 pairs. Three real re-reads — every one of them already caught by the existing requery route via structural + sha256. The other three were the same PDF retried three times in one session. Zero new signal.
The narrower detector would have shipped duplicate output for cases already handled. The pre-registration held. The detector stayed dead.
Full write-up — predictions and results in one file, timestamped, so you can see what was pre-committed vs what was measured: REREAD_DETECTOR_PREREG.md
What Clew is and isn't
It diagnoses. It does not fix. The output is a Markdown report you read. What to change in your agent — prompt, cache placement, tool routing — is a call only you can make.
One working pattern, on purpose. repeat / requery. A second pattern (pingpong, reasoning-level — the agent-to-agent thing I originally set out to find) is implemented but has never fired on a real trace format I've validated. I don't advertise what I haven't observed.
Reads what you already have. CC session JSONL, OpenTelemetry SDK-JSON, OpenInference (Phoenix / TRAIL), Toolathlon, RedundancyBench. Cursor and Codex not yet — their local formats are under evaluation.
No measured savings yet. I can detect and estimate. I have zero before/after data from someone who fixed something and watched their bill drop.
239 tests, CI on every PR, frozen params enforced as failing tests. Changing phi requires a documented recalibration, not a nudge.
Try it
pip install "clew-custos[detect]"
python -m clew analyze ~/.claude/projects//.jsonl --out report.md
The bare name clew on PyPI is an unrelated placeholder — install clew-custos. The module still imports as clew.
Repo: https://github.com/JEONSEWON/Clew-by-Custos
What I'd like to know
If you've run agents with a large MCP tool surface — 100+ tools, mixed vendors — and you have a trace archive lying around, does the tool count correlate with your duplicate-call rate? My Toolathlon-vs-CC delta is one data point. Not a law. If your numbers disagree with mine, I'd rather know.
Top comments (0)