A pilot benchmark, a $5.60 receipt, and a 97%-savings number that was actually a silent failure.
How it started
This started from a plain problem: I kept hitting token limits at work. I was using Claude Code for real engineering work, and I was burning through budget faster than I wanted. The obvious question was: can I cut that down without hurting the quality of what the agent produces? Not "just use a cheaper model and hope." Something more deliberate — a set of tools that each attack a different part of the token bill. How much context gets read. How much gets re-read. How verbose the agent's own output is. How it finds its way around a codebase in the first place.
That question turned into a side project: token-optimization-stack, a public repo with setup docs for tools that reduce token spend. And token-stack-benchmarks, a benchmark harness to actually test whether any of it worked.
I'm writing this up because the project ended, a few weeks in, in a place I didn't expect. Not with a working stack and a savings number. Instead, with proof that the token-savings numbers I was looking at were actively misleading — and a cost problem that made the whole thing stop making sense before I could even publish a result. I think both of those are more useful to write about than a clean win would have been.
What the stack looked like early on
The first version of the stack had five tools in it:
- Graphify — turns a codebase into a queryable knowledge graph. The agent can ask "what calls this" instead of reading files to find out.
- Serena — lets the agent navigate and edit code by symbol, instead of raw file reads and text edits.
- Headroom — advertised as transparent context compression. Its docs said it needed "no behavioral changes" once installed.
- LiteLLM — a routing layer. The idea: send easy subtasks to a cheap model and hard ones to an expensive model.
- Caveman — compresses the agent's own output. Terser replies, compressed subagent output, less back-and-forth.
Two of those five didn't survive contact with a real benchmark.
Headroom didn't do what its own docs implied. The only way to register it without wrapping the whole claude command in a separate launcher is headroom init claude. That just adds an on-demand MCP tool — something the agent can call, not something that compresses context automatically. Running headroom doctor confirmed nothing was actually being routed through it unless you also ran a separate proxy process with an ANTHROPIC_BASE_URL override. That's a much heavier setup than "no behavioral changes" suggested. On top of that, its mcp serve command crashed against a current MCP SDK — it needed an old, pinned mcp<2 dependency just to start.
LiteLLM had a different problem. Its usage-based routing is a load-balancing strategy across provider endpoints, not the complexity-based, per-task routing I actually wanted. And mechanically, Claude Code sends one fixed model for an entire session — there's no way to swap models mid-task based on how hard a step is. The tool I wanted didn't exist yet, at least not in this shape.
I removed both rather than keep them in as unverified claims. What was left — Graphify, Serena, a compression/caching layer called LeanCTX, and Caveman — became the actual stack I tested.
Why I had to stop
Not because the idea was wrong. That came later. The experiment itself stopped making financial sense.
The rigorous version of this test — real tasks from SWE-bench Verified and Multi-SWE-bench, sixteen repos, five versions of the stack, three repeats each — works out to about 4,800 agent runs. I never got close to that. Instead I ran a much cheaper pilot: 31 tasks, 2 versions of the stack, one repeat, on the cheapest model I had (claude-haiku-4-5, medium effort). Even that only partly finished — 11 of 31 task pairs — and it already cost about $5.60 in raw API spend. That's before EC2 costs, Docker builds, or the multi-day slog of getting this running cleanly on both EC2 and an Apple Silicon Mac.
Scale that same per-run cost up to the full test and you pass $1,200 in API spend — on the cheapest model available, before a single result is even trustworthy. Sonnet costs 2x what Haiku does, on both input and output tokens ($2/$10 per million tokens vs. Haiku's $1/$5). So switching to it to get a trustworthy result would push the same test past $2,400. And Haiku wasn't trustworthy: it got zero correct fixes on the Java tasks, and it broke two of the three Python tasks the plain baseline had already solved.
Is this savings number — or this correctness failure — actually about the stack? Or is it about the fact that I'm running everything on the cheapest model I could afford to run 4,800 times of?
I didn't have a good answer. That's where I stopped.
What I actually found, for what it's worth
Even the partial pilot data was worth sharing, because it directly contradicts what a token-savings-only view would have told me. On the three Python tasks the plain baseline agent solved correctly, adding the full stack did this:
- On one task, the agent treated a clear, self-contained bug report as if it were ambiguous. It asked a one-line clarifying question on its very first turn, then just stopped —
num_turns: 1, no error, nothing left to score. The plain baseline took 30 turns on the exact same prompt and fixed the bug. Read only off the token dashboard, this run showed 97% fewer tokens used — the single best-looking number in the whole pilot, and it came from the one run that did no work at all. - On a second task, the stack produced a real patch. It applied cleanly. The target test still failed.
- The third task stayed correct — but used more tokens than the baseline, not fewer.
I'm not treating "2 of 3" as a rate. Three tasks is too small a sample to turn into a percentage. But something else holds, even at this size: the token numbers and the correctness numbers pointed in opposite directions, and the worst result in the batch produced the best-looking number. That doesn't need a bigger sample to be true — it happened, on a real task, and it's exactly the kind of failure a token-savings-only report can't catch.
I'd also expect this to get worse on a weak model, not better. Haiku has less room to recover once a terser style takes away its ability to push back or think through whether a task is really ambiguous. A stronger model might ask the same question but keep working anyway — or not need to ask at all. I didn't get to test that. It's a specific, checkable prediction for whoever picks this up next, not just a guess.
Token and cost savings numbers, without a real correctness check against the actual test suite, aren't just incomplete — they can point in exactly the wrong direction. And the biggest, flashiest savings number is a plausible place for that to happen, not an unlikely one.
The fix is simple: report cost per solved task, not cost per task. Under that measure, the 97%-savings run isn't a win with an asterisk. It's infinitely expensive, because it solved zero tasks. That one change closes the trap — a dashboard built around it can't turn a silent failure into a headline number.
Pair that with something even cheaper to check: turn count. A run that takes 1 turn when the baseline took 30 is a giant red flag, one that no token dashboard shows on its own. And unlike correctness scoring, checking it costs nothing — no test suite, no scoring setup, no Docker. It's already sitting in the same log that produced the token numbers.
If you want to pick this up
I'm not going to keep running this. Not because I think the question is answered — I just can't afford to answer it properly right now. If you want to take it further, both repos are public:
- token-optimization-stack — the stack itself: setup scripts and docs for Graphify, Serena, LeanCTX, and Caveman, plus the benchmark methodology this pilot followed.
- token-stack-benchmarks — the test harness: a Dockerized runner for each version of the stack, task sampling, and the SWE-bench / Multi-SWE-bench scoring scripts. Also everything I ran into getting a Linux-shaped harness to work on both EC2 and an Apple Silicon Mac — case-sensitivity bugs, CPU architecture mismatches, a new Python version breaking a scoring dependency, and more.
Contributions and forks are welcome. So are "here's why your pilot was wrong" pull requests. A few concrete places to start:
- The broken-patch case is still a mystery. The empty-patch failure has a clear cause now (see above). This one doesn't. On the second Python task, a real patch applied cleanly and still didn't fix the bug. Nothing in the logs explains why the stack produced a wrong-but-plausible answer instead of a right one. That's the harder failure mode, and nobody's looked into it yet.
- Running the full 5-arm test would confirm a real suspect, not just a guess. The 97% run failed because of behavior, not because context got lost. That points at Caveman specifically, and mostly clears LeanCTX, Graphify, and Serena for that run. Running the full test would show whether that holds up, or whether it was a one-off.
- The haiku-weakness prediction above is easy to test. Run the same pilot on Sonnet or Opus and see if the correctness problem gets smaller. That's a real experiment you can run — not just "try a bigger model and see."
Top comments (1)
The part I trust most here is that the best-looking savings number came from the run that damaged correctness. Token reduction is easy to measure and very easy to optimize into the wrong target. The useful benchmark is cost per accepted fix, not tokens per attempt.