To save tokens, I capped every response from my coding agent's code-graph tool at 12,000 characters. One answer then took 18 turns instead of 7, and the input tokens it was billed for went from 267k to 708k, 2.65 times as many.
This post explains why a smaller response used more tokens, and what else I found when I rebuilt the measurement on public repositories: 252 headless Claude Code runs (non-interactive, one question each), every token counted. You'll also get the benchmark's exact command line and a script that shows how much of your own sessions' input comes from the prompt cache. The runner and every raw run, answers included, are in claude-code-token-bench.
Why I was cutting tool output
I wanted to know whether a code graph makes Claude Code, Anthropic's coding agent, cheaper than plain search. The graph came from codebase-memory-mcp, an open-source MCP server that builds a graph of a repository's symbols and the calls between them. MCP, the Model Context Protocol, is how Claude Code plugs in outside tools.
The first measurement went badly for the graph. On five questions, the agent with the graph was billed 2.35 times the input of the agent with grep, 4.26M tokens against 1.81M, and it came out cheaper on only one question.
So I looked at what the server sent back. One search hit was 1,342 bytes of JSON, most of it hash vectors, complexity metrics and a list of every identifier in the function's body. The same hit as a grep line was 115 bytes. On top of that, the server's 14 tool definitions rode along in every prompt.
I put a thin MCP server of my own in front of it, graph-ask. This wrapper dropped the fields no code question needs, which brought a hit down to 251 bytes. It also capped every response at 12,000 characters: it halved the longest list until the response fit, and noted how much it had cut.
What the cap did
The tool that returns a repository's structure hit the cap. The model got a shortened list with a note saying how much was missing, and asked again. That answer took 18 turns where it had taken 7, and its billed input rose from 267k tokens to 708k.
A turn is one round trip to the model. Each one is a new API request, and it carries the whole conversation: the system prompt, the tool definitions, the question and every tool result so far. Divide the totals by the turns and the average turn comes to about 38,000 tokens before the cap and 39,000 after. The smaller responses didn't make a turn any cheaper. The turn count went up 2.57 times and the tokens 2.65 times, which came to 441k more tokens in all.
That is also why a turn costs more the later it comes. Each tool result joins the history that every later turn sends again, so billed input grows faster than the number of turns.
One limit on the 2.65: in that round I counted every input token the same, including the ones served from the prompt cache, the unchanged start of the conversation that Anthropic keeps between requests. Anthropic bills those cache reads at a tenth of the input price, so 2.65 is a ratio of tokens, not of dollars. The public benchmark below prices each kind of token at its own rate.
What I changed in the wrapper
I kept the trimming and rebuilt the rest around the number of turns. One tool call now answers a whole structural question, with a function's definition, source, callers and callees together, so the model doesn't assemble the answer over several turns. A script summarizes each repository's layout once into a file, and the agent reads that file instead of querying the graph. Asked for a project's HTTP routes, the tool returns each handler along with the config files that hold the URL prefixes. And the cap went up to 60,000 characters.
Together those changes took that round's questions from 83 turns and 3.46M billed input tokens to 36 turns and 1.62M. Listing a project's HTTP routes alone went from 23 turns to 8.
The current version also checks a name before it queries. The graph matches names exactly, so a near miss returns nothing and costs a turn to recover. When the name doesn't exist, the tool takes the top search hit and says it did.
The wrapper still labels a cut when a response hits the cap. From server.mjs, shortened:
// Bring a response under MAX_CHARS by shortening its longest arrays, halving the
// worst offender at a time, and recording what was dropped next to it. Slicing the
// serialized JSON instead would hand the model invalid syntax and no idea what is
// missing — the truncation has to stay legible, or it reads as a complete answer.
const kept = worst.arr.slice(0, Math.max(1, Math.floor(worst.arr.length / 2)));
parent[key] = kept;
parent[`${key}_truncated`] = `showing ${kept.length} of ${original}`;
The 12,000-character version had the same label, and the model went back for the rest anyway. The label stays so that a cut list never passes for the complete answer.
Bash and subagents hid more turns
The cap wasn't the only thing adding turns in that round. The grep-only setup was allowed Bash and subagents, the helper agents Claude Code can start for a subtask, and the model ran grep | head loop after loop, much of it inside subagents. It was billed 1.8M input tokens. With Bash and subagents denied, the same setup needed 252k.
Two things kept this out of sight. Claude Code's turn count covers only the main loop, so a subagent's turns are billed but missing from the count of the run you're watching. And scoped Bash permissions don't match piped commands: a grant for grep alone doesn't cover the pipelines the model writes, so different setups hit different numbers of permission denials.
A denial can even look like a saving. In headless mode nobody is there to approve a tool, so an MCP server's tools have to be listed in --allowedTools, or every call is refused while their definitions are still paid for. My very first graph run looked six times cheaper than grep. In that run the graph tools had been refused.
For read-only questions the benchmark gave the agent only Read, Grep and Glob, Claude Code's built-in tools for opening a file, searching file contents and finding files by name. Bash was denied, and so was Agent, the tool that starts subagents. For an agent that only has to read code, that's the set I'd start from. The benchmark ran each question like this, from the repository's checkout:
claude -p "Which functions or route files call createPaymentIntent? Ignore tests. Give the file:line of each call site, nothing else." \
--output-format stream-json --verbose \
--strict-mcp-config --setting-sources project --no-session-persistence \
--allowedTools Read Grep Glob \
--disallowedTools Bash Agent
--strict-mcp-config loads only the MCP servers named on the command line, none here. --setting-sources project skips user-level settings, hooks and plugins, so nothing installed on the machine steers the model. stream-json reports each tool call and the run's token usage, which is what the benchmark recorded.
The earlier result didn't survive public repositories
That earlier round had also found my wrapper 26% cheaper than grep with Claude Haiku and 18% with Claude Sonnet. But it had given a routing rule, a line in the prompt that says when to use the graph and when to use grep, to every setup, including the one with no graph at all. And its repositories were private, so no one else could check it. I rebuilt the benchmark on public repositories, and that result did not reproduce.
The setup: Claude Code 2.1.283 in headless mode with Claude Haiku 4.5 and Claude Sonnet 5. Three small repositories have 1.6k to 5k graph nodes each, where a node is one function, class, file or similar item: FastAPI's full-stack template, ltx-2-mlx (an MLX port of the LTX-2 video model) and avoid-ai-writing (a tool that flags AI writing patterns). One large repository, dub, a link attribution platform, has 25k. The three small repositories share twelve questions, and dub has twelve of its own. In each set, six are about structure, such as who calls a function or the path from an endpoint down to the database write, and six are about exact strings, such as a config value or an error message. Every answer is checked for the facts it must contain. The 252 runs cost $14.69 at API prices.
Tokens are priced by kind, with Anthropic's multipliers: fresh input at 1, a cache write at 1.25 and a cache read at 0.1. The setups run in a different order on each repetition, only the setups with a graph get the routing rule, and none of them has Bash or Agent.
Median priced tokens per run, and correct answers:
| repositories, model | graph through my wrapper | graph server directly | Read and Grep only |
|---|---|---|---|
| 3 small, Haiku 4.5 | 30,321 · 28/36 | 43,846 · 32/36 | 31,159 · 33/36 |
| dub, Haiku 4.5 | 18,791 · 22/24 | 20,687 · 22/24 | 14,955 · 20/24 |
| dub, Sonnet 5 | 17,227 · 22/24 | 23,085 · 22/24 | 13,348 · 22/24 |
The graph never beat Read and Grep on tokens. With Sonnet on dub, Read and Grep were cheaper on 11 of the 12 questions at the same accuracy (p = 0.034, a Wilcoxon signed-rank test over per-question medians). My wrapper did beat the bare graph server in all three settings, on 8, 11 and 9 of the 12 questions.
The graph's one clear win was accuracy. On dub, Haiku answered the structural questions correctly 12 times out of 12 with it and 8 out of 12 without, at a median of 23.5k tokens against 16.0k on those questions.
The most expensive setup was a graph the agent mostly ignored
By default, Claude Code 2.1.283 lists an MCP server's tools by name and loads a tool's full definition only when the model asks for it through its ToolSearch tool. In headless mode Haiku mostly didn't ask. On the twelve small-repository questions:
| setup | runs that called the graph | correct | median priced tokens |
|---|---|---|---|
| definitions deferred (the default), routing rule in the prompt | 2/12 | 11/12 | 41,008 |
| all definitions loaded, no rule | 4/12 | 9/12 | 62,008 |
| all definitions loaded, with the rule | 22/36 | 28/36 | 30,321 |
The first two rows are one run per question; the last is the main run, three per question. Setting ENABLE_TOOL_SEARCH=false in the environment makes Claude Code load every definition up front. In six pilot runs with the default, the graph was never called. With that variable set, the same question went through the graph in three turns.
Loaded definitions are paid for on every turn, whether anything calls them or not. That is why the mounted graph with no rule cost the most of any setup here. Even with the rule, Sonnet picked the graph in 10 of its 24 runs on dub and Grep for the rest.
If you keep an MCP server mounted, load its definitions and tell the agent when to use it. The benchmark passed the rule with --append-system-prompt. Day to day it belongs in CLAUDE.md, the instruction file Claude Code reads at the start of a session:
Structure (who calls what, call chains, routes, module layout, a symbol end to end): one graph_ask call.
Exact strings, config values, flags, error text: Grep and Read.
graph_ask is my wrapper's one-call tool; name your own server's tool there. If you can't say when the agent should use a server, unmount it.
A warm cache made later setups look cheaper
Back-to-back runs on the same repository share a prompt cache. When the setups ran in a fixed order, the later ones started warm and got a discount the first one never had, which is why the order now rotates.
You can see how much the cache does in your own sessions. tools/cache-audit.mjs in the repository reads Claude Code's local session logs, reports what share of billed input came from the cache, overall and per project, and lists the worst sessions:
node tools/cache-audit.mjs --days 7
In my earlier benchmark runs, 83 to 88 percent of billed input came from the cache. A much lower share points at something that changes the start of the prompt between turns, like a timestamp in a system prompt or a tool set that shifts.
Where a graph does pay
A second benchmark ran in GitHub Copilot CLI on two much bigger repositories, kubernetes (146k nodes) and vscode (219k). There the graph cut Haiku's priced tokens on structural questions to 0.29 of grep's, and it was cheaper on 15 of those 16 questions. Sonnet's went to 0.74 and Opus 5.5's to 0.97. The details are in Where a code graph pays off. In Claude Code I haven't tried repositories that large, or Opus.
What these runs can't tell me
Every run in this post was a single question in a fresh session. In a long interactive session two things change, and they pull in opposite directions: each extra turn sends a much longer history again, and most of that history is read from the cache at a tenth of the price. So in a long session, does a tight cap on tool output cost more than the 2.65 times I measured here, or less?
Top comments (0)