DEV Community

Cover image for Stop Counting Tools. Start Counting Turns.
AheadKit
AheadKit

Posted on

Stop Counting Tools. Start Counting Turns.

Two days ago I measured what 200 tools cost an agent before it does any work: 24,522 tokens per message, 122.6 per tool, a straight line with no bulk discount. The comments on that piece kept asking the same follow-up, and paul_irolla asked it sharpest: per-message tokens are the menu, not the bill. The bill is per solved task, and the schema goes back over the wire on every turn, so its cost is size times turns. He measures that on THOL, the leaderboard he maintains, and he pointed me at its published per-run data. So I took his data, all 4,310 runs of it, and sliced it myself by tool-list size instead of trusting the board's grouping.

The first thing I found undoes half the question. There is no 200-tool entry in THOL. The biggest real tool list anyone measured there is 30 tools. My 200-tool number was a synthetic server I built in the morning; nobody in this data runs anything like it. So this piece answers the question the data can actually answer: what does a list of 0, 8, 13 and 30 tools cost per solved task, on the same tasks, same model, same board? And the answer argues with the piece I wrote two days ago.

My claim is this: the size of the list does not decide the bill. What the tools do does. A 30-tool list costs about the same per solved task as no list at all. A 13-tool list costs 41% more. If you are counting tools to predict your bill, you are counting the wrong thing.

The numbers

THOL publishes every run: the task, the tool, whether it solved it, the tokens, the turns, the cost. The board groups rows by how big the run was, not by how many tools the list held, which is why paul_irolla said to slice it ourselves. The only list-size field in the data is the count of tools each tool's own server exposes, and only three of the thirteen entries expose a server at all: codegraph with 8 tools, lean-ctx with 13, code-review-graph with 30. Everything else works through a hook, a proxy, a prompt or a command line, and vanilla Claude Code, the control, runs with none. I sliced the board's own headline set, the 17 long tasks where a run burns more than 200,000 tokens, on the campaign the board's own table uses, 170 runs per entry, the same 17 tasks for all four. Then I ran the same slice over both campaigns in the file as a check, and the shape held.

List Tools Tokens per solved task Turns per solved task Cost per solved task
Vanilla Claude Code 0 301,467 16.95 $0.5426
codegraph 8 294,318 17.60 $0.5367
code-review-graph 30 304,341 17.80 $0.5221
lean-ctx 13 425,411 22.43 $0.5614

Read the first three rows twice. Zero tools, 301,467 tokens per solved task. Eight tools, 294,318, down 2.4%. Thirty tools, 304,341, up 1.0%. The list nearly quadruples and the bill does not move. Then the 13-tool list arrives at 425,411, up 41%, with five and a half extra turns per solved task. The expensive list is not the big one. It is the middle one, and the reason is not its size.

One more check before the argument, because means lie. The median run tells a slightly different story: vanilla 167,766 tokens, codegraph 191,700, code-review-graph 198,273, lean-ctx 310,098. On medians the two lists cost 14% and 18% more than vanilla, not nothing, and lean-ctx is 85% more. But the board's own noise floor says a run varies by about 17% from one attempt to the next, so a gap of a few percent between rows is not a ranking, it is weather. The lean-ctx gap is far outside that weather. The other two are inside it.

Problem 1: the rent is real, and it is small next to the run

Two days ago I showed the rent: 122.6 tokens per tool, per message. At 30 tools that is about 3,700 tokens a message. Over 17 turns that is roughly 60,000 tokens of rent inside a 304,000-token run, and most of it arrives as cached re-reads, billed cheap. At 8 tools it is about 1,000 a message, 17,000 over the run. The difference between the two lists is real money, a few cents a task, and it vanishes inside the noise of a run that burns 300,000 tokens. The rent matters, but only when the list gets big enough to matter, and nobody in this data runs a list big enough. That is the honest reconciliation with my own earlier piece: the per-message line is straight and true, and at the sizes anyone actually runs, the run itself is the bill.

Where do those 300,000 tokens actually go? Barely any of it is fresh input, about 0%, and barely any of it is output, about 2%. Between 90% and 92% of it is cache re-reads: the model reading the same growing context back on every turn. Every extra turn re-reads everything that came before it. That is why the turn count, not the list size, is the number to watch.

Problem 2: the most-used tool is the most expensive one

Lean-ctx is the one tool in the whole board the agent actually calls. In 154 of its 170 runs, about 11 calls a run. Every other entry sits near zero: codegraph's tools were called in 4 of 170 runs, code-review-graph's in none at all. And lean-ctx still came out the most expensive, by a distance. Being used is necessary, the board's own write-up says so, but it is not sufficient. What the tool returns has to be worth more than the tokens it costs to offer it and to read it, and when the returns are poor the agent falls back to its normal way of working, re-reading and re-searching, which costs turns. Lean-ctx paid the adoption price and then paid the fallback price on top. Eleven calls a run, five and a half extra turns, 41% more tokens. Use is not a win. Use that works is a win.

Problem 3: the list nobody calls costs nothing

Code-review-graph is my favourite row in the whole file, because it is the cleanest test in it. Thirty tools offered, never called once, in 170 runs. Its bill on the mean is 1% above vanilla, inside the noise. That is the pure standing cost of a 30-tool list: about nothing. So the fear that kept me counting tools, that merely offering a long list would quietly tax every turn, does not survive contact with this data, at 30 tools at least. The list you never use costs nothing. The list you use badly costs everything. The difference is not the count. It is the call.

Problem 4: per solved task is nearly per run here, and that is a caveat, not a trick

One more honest note, because the denominator matters. Of the 4,310 runs in the file, 4,309 solved their task. One failed. So "per solved task" is effectively "per run" in this data, and the solved-task framing barely moves any number. That is not a flaw in the slice, but it is a limit on what it proves: these tasks are near-always solved by this model, so the data cannot tell you what a hard task costs when the agent fails twice before succeeding. The per-solved-task bill on genuinely hard work is still an open question, and it is a better one than the one I answered.

What would change my mind: a real 200-tool entry in this data, or any data. At 200 tools the rent stops being a rounding error, about 24,500 tokens a message, 17 turns, and the standing cost alone would be a six-figure token pile before the agent does anything. My earlier piece measured that rent on a synthetic server. This piece shows that nobody has published a real run at that size. Until someone does, "big lists are expensive" is a true statement about a menu nobody has ordered from.

If you run agents with a long tool list, what does your bill actually track, the list size, the turns, or something else entirely? And what is the biggest list you have run in real work, not in a test?


P.S. Everything I write stays free — that part doesn't change. But if this one was worth something to you, you can buy me a coffee ☕ (one-time or monthly, both work). It covers the API credits and hosting behind these experiments — and honestly, it's also the nudge that gets the next one started on the evenings I'd otherwise skip it. — Salam

Buy me a coffee

AheadKit on Gumroad — Beginner-Friendly Books & Packs

Top comments (0)