A pilot benchmark, a $5.60 receipt, and a 97%-savings number that was actually a silent failure.
How it started
This started from a plain problem: I kept hitting token limits at work. I was using Claude Code for real engineering work, and I was burning through budget faster than I wanted. The obvious question was: can I cut that down without hurting the quality of what the agent produces? Not "just use a cheaper model and hope." Something more deliberate — a set of tools that each attack a different part of the token bill. How much context gets read. How much gets re-read. How verbose the agent's own output is. How it finds its way around a codebase in the first place.
That question turned into a side project: token-optimization-stack, a public repo with setup docs for tools that reduce token spend. And token-stack-benchmarks, a benchmark harness to actually test whether any of it worked.
I'm writing this up because the project ended, a few weeks in, in a place I didn't expect. Not with a working stack and a savings number. Instead, with proof that the token-savings numbers I was looking at were actively misleading — and a cost problem that made the whole thing stop making sense before I could even publish a result. I think both of those are more useful to write about than a clean win would have been.
What the stack looked like early on
The first version of the stack had five tools in it:
- Graphify — turns a codebase into a queryable knowledge graph. The agent can ask "what calls this" instead of reading files to find out.
- Serena — lets the agent navigate and edit code by symbol, instead of raw file reads and text edits.
- Headroom — advertised as transparent context compression. Its docs said it needed "no behavioral changes" once installed.
- LiteLLM — a routing layer. The idea: send easy subtasks to a cheap model and hard ones to an expensive model.
- Caveman — compresses the agent's own output. Terser replies, compressed subagent output, less back-and-forth.
Two of those five didn't survive contact with a real benchmark.
Headroom didn't do what its own docs implied. The only way to register it without wrapping the whole claude command in a separate launcher is headroom init claude. That just adds an on-demand MCP tool — something the agent can call, not something that compresses context automatically. Running headroom doctor confirmed nothing was actually being routed through it unless you also ran a separate proxy process with an ANTHROPIC_BASE_URL override. That's a much heavier setup than "no behavioral changes" suggested. On top of that, its mcp serve command crashed against a current MCP SDK — it needed an old, pinned mcp<2 dependency just to start.
LiteLLM had a different problem. Its usage-based routing is a load-balancing strategy across provider endpoints, not the complexity-based, per-task routing I actually wanted. And mechanically, Claude Code sends one fixed model for an entire session — there's no way to swap models mid-task based on how hard a step is. The tool I wanted didn't exist yet, at least not in this shape.
I removed both rather than keep them in as unverified claims. What was left — Graphify, Serena, a compression/caching layer called LeanCTX, and Caveman — became the actual stack I tested.
Why I had to stop
Not because the idea was wrong. That came later. The experiment itself stopped making financial sense.
The rigorous version of this test — real tasks from SWE-bench Verified and Multi-SWE-bench, sixteen repos, five versions of the stack, three repeats each — works out to about 4,800 agent runs. I never got close to that. Instead I ran a much cheaper pilot: 31 tasks, 2 versions of the stack, one repeat, on the cheapest model I had (claude-haiku-4-5, medium effort). Even that only partly finished — 11 of 31 task pairs — and it already cost about $5.60 in raw API spend. That's before EC2 costs, Docker builds, or the multi-day slog of getting this running cleanly on both EC2 and an Apple Silicon Mac.
Scale that same per-run cost up to the full test and you pass $1,200 in API spend — on the cheapest model available, before a single result is even trustworthy. Sonnet costs 2x what Haiku does, on both input and output tokens ($2/$10 per million tokens vs. Haiku's $1/$5). So switching to it to get a trustworthy result would push the same test past $2,400. And Haiku wasn't trustworthy: it got zero correct fixes on the Java tasks, and it broke two of the three Python tasks the plain baseline had already solved.
Is this savings number — or this correctness failure — actually about the stack? Or is it about the fact that I'm running everything on the cheapest model I could afford to run 4,800 times of?
I didn't have a good answer. That's where I stopped.
What I actually found, for what it's worth
Even the partial pilot data was worth sharing, because it directly contradicts what a token-savings-only view would have told me. On the three Python tasks the plain baseline agent solved correctly, adding the full stack did this:
- On one task, the agent treated a clear, self-contained bug report as if it were ambiguous. It asked a one-line clarifying question on its very first turn, then just stopped —
num_turns: 1, no error, nothing left to score. The plain baseline took 30 turns on the exact same prompt and fixed the bug. Read only off the token dashboard, this run showed 97% fewer tokens used — the single best-looking number in the whole pilot, and it came from the one run that did no work at all. - On a second task, the stack produced a real patch. It applied cleanly. The target test still failed.
- The third task stayed correct — but used more tokens than the baseline, not fewer.
I'm not treating "2 of 3" as a rate. Three tasks is too small a sample to turn into a percentage. But something else holds, even at this size: the token numbers and the correctness numbers pointed in opposite directions, and the worst result in the batch produced the best-looking number. That doesn't need a bigger sample to be true — it happened, on a real task, and it's exactly the kind of failure a token-savings-only report can't catch.
I'd also expect this to get worse on a weak model, not better. Haiku has less room to recover once a terser style takes away its ability to push back or think through whether a task is really ambiguous. A stronger model might ask the same question but keep working anyway — or not need to ask at all. I didn't get to test that. It's a specific, checkable prediction for whoever picks this up next, not just a guess.
Token and cost savings numbers, without a real correctness check against the actual test suite, aren't just incomplete — they can point in exactly the wrong direction. And the biggest, flashiest savings number is a plausible place for that to happen, not an unlikely one.
The fix is simple: report cost per solved task, not cost per task. Under that measure, the 97%-savings run isn't a win with an asterisk. It's infinitely expensive, because it solved zero tasks. That one change closes the trap — a dashboard built around it can't turn a silent failure into a headline number.
Pair that with something even cheaper to check: turn count. A run that takes 1 turn when the baseline took 30 is a giant red flag, one that no token dashboard shows on its own. And unlike correctness scoring, checking it costs nothing — no test suite, no scoring setup, no Docker. It's already sitting in the same log that produced the token numbers.
If you want to pick this up
I'm not going to keep running this. Not because I think the question is answered — I just can't afford to answer it properly right now. If you want to take it further, both repos are public:
- token-optimization-stack — the stack itself: setup scripts and docs for Graphify, Serena, LeanCTX, and Caveman, plus the benchmark methodology this pilot followed.
- token-stack-benchmarks — the test harness: a Dockerized runner for each version of the stack, task sampling, and the SWE-bench / Multi-SWE-bench scoring scripts. Also everything I ran into getting a Linux-shaped harness to work on both EC2 and an Apple Silicon Mac — case-sensitivity bugs, CPU architecture mismatches, a new Python version breaking a scoring dependency, and more.
Contributions and forks are welcome. So are "here's why your pilot was wrong" pull requests. A few concrete places to start:
- The broken-patch case is still a mystery. The empty-patch failure has a clear cause now (see above). This one doesn't. On the second Python task, a real patch applied cleanly and still didn't fix the bug. Nothing in the logs explains why the stack produced a wrong-but-plausible answer instead of a right one. That's the harder failure mode, and nobody's looked into it yet.
- Running the full 5-arm test would confirm a real suspect, not just a guess. The 97% run failed because of behavior, not because context got lost. That points at Caveman specifically, and mostly clears LeanCTX, Graphify, and Serena for that run. Running the full test would show whether that holds up, or whether it was a one-off.
- The haiku-weakness prediction above is easy to test. Run the same pilot on Sonnet or Opus and see if the correctness problem gets smaller. That's a real experiment you can run — not just "try a bigger model and see."
Top comments (10)
Killing the project and publishing the kill is worth more than most clean wins - and your 97%-run is the sharpest example of a metric without a negative control I've seen written up: the number was never asked "what do you show when nothing useful happened?" We fell into the same trap from another side once: a savings dashboard of ours counted heartbeat pings into the total, so the number climbed while the system did nothing of value. The fix was the same shape as yours - change the denominator to real, solved work only. Cost per solved task doesn't just close the trap, it survives scaling; tokens per attempt never does.
On your unsolved mystery - the clean-applying patch that still failed: one checkable suspect is the stack's view of the codebase. Graphify/Serena show the agent symbols instead of raw files, and a symbol view can hide a second call site or a data path that a plain file read would have dragged into context. The cheap test is already in your logs, like the turn count was: diff the set of files/symbols each run actually READ, baseline vs. stack, on that task. If the stack run never saw the code region the fix needed to touch, you've found the mechanism - and it would mean the failure belongs to the navigation layer, not to Caveman's terseness. Did the logs keep the read-trace, or did that get discarded with the token counts?
Vinh's cheap-gate suggestion above matches what we ended up building for our own benchmark harness: every arm first passes three gates that cost nearly nothing - unsolved must fail, solved must pass, and a known-bad probe must go red. A run with no scorable artifact never reaches the expensive stage. Happy to compare notes if you pick this up again - your "cost per solved task" line is going straight into how we report our own numbers, with your name on it.
Thank you! This is the kind of feedback that's actually worth more than the post itself. Glad the denominator fix is useful on your end — happy to have my name on it.
The read trace, it wasn't kept :( The harness calls claude -p --output-format json, which only returns the final summary (turn count, token usage, cost, result text) — not a per-turn tool-call transcript. So right now your suspect (symbol view hiding a call site the baseline's raw reads would've caught) is checkable in principle, but not from data I already have — it needs a rerun with a transcript captured on purpose.
I am glad it converges with what you already ship. If I pick this back up, I'll take you up on comparing notes.
One flag saves the rerun from becoming a harness rebuild: claude -p --output-format stream-json emits per-event output including every tool call - the per-turn transcript you're missing is a capture setting, not missing data. If you pick it back up, the notes-comparison stands.
That is a good point. I have aded it to the benchmarking guide. If I or anyone else who picks it up will add that option and try the runs again.
The part I trust most here is that the best-looking savings number came from the run that damaged correctness. Token reduction is easy to measure and very easy to optimize into the wrong target. The useful benchmark is cost per accepted fix, not tokens per attempt.
True. Token optimisation should not come at the cost of correctness. Worst case I can image is exploring an unfamiliar project and getting the wrong info.
The 4,800-run estimate assumes every arm deserves the expensive correctness stage. A cheaper first gate can reject runs with no patch, a one-turn stop, or a test suite that never executes, then spend repeats only on arms that produce a scorable artifact. It will not make the pilot statistically conclusive, but it prevents the 97%-savings failure from consuming the same benchmark budget as a surviving arm.
Good catch, and you're right that I priced the full pipeline uniformly. It does cut the part of the cost that was pure waste (scoring runs that were never going to produce a result), so the real budget goes further toward the arms that actually need it.
Adding this to the open list for whoever picks up the harness next.
The 97% savings number that came from a quit after one turn is the exact trap I keep seeing when people budget coding agent runs. Token dashboards reward silence and early exits the same way they reward real compression, so the best looking row is often the run that never opened the bug. Stopping once Haiku made the correctness signal unreadable feels like the honest call. Curious whether you would ever rerun a tiny pilot on a stronger model with fewer tasks just to see if the stack moves patch quality, or whether the cost floor already closed that door.
To be very honest I did run a tiny pilot on Sonnet 5 high effort. It worked as well, but I did the maths and the cost to keep it running came up too high. I think I should publish the results from that run as well. I will dig them up and put them up as well.