A data point from a real server, counted your way (cl100k, compact JSON). Disclosure: I maintain CutVoke, an open-source video editor that agents drive over MCP.
Its server lists 52 tools, which come to 7,958 tokens, about 153 per tool. That's heavier than your synthetic ones, partly because the descriptions are in Chinese. Your whitespace point holds too: the same 52 pretty-printed are 12,567.
The bigger number is what isn't on the menu. There are 110 edit commands, and their payload schemas together come to 35,965 tokens. Instead of 110 more tools they all go through one command_apply(type, payload), and the agent fetches the schema for the one command it needs with commands_list(type=...) or editor_help(topic="command", command=...) right before using it. Even the name-only catalog (5,924 tokens) is fetched on demand rather than listed.
The trade-off: an unfamiliar command costs an extra round trip, and the model only sees a schema after it asks. An unknown command name comes back with an error pointing to commands_list. So your "let the agent ask for more" also works inside a single server, not only by splitting servers.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
This is the number I was missing — 153 a tool on a real server sits right between my lean synthetic ones and the filesystem server's 263, and the Chinese descriptions explain part of it. But the 35,965 hiding behind the lazy fetch is the part that got me: the menu looks cheap because the real schemas only show up when the agent asks. That trade-off's real too, an extra round trip every time the model meets a command it doesn't know. You basically built the version of 'let the agent ask for more' that lives inside one server instead of across several.
Yes, that round trip is the part I keep weighing. Two things in the server try to make it land at the right moment: the MCP instructions tell the agent to call editor_help(topic='command', ...) or commands_list before an unfamiliar edit, so the lookup happens up front rather than after a failed call, and editor_help(topic='recipes') has worked examples with the payload shapes for common edits. It doesn't remove the extra call, it just makes it predictable. And yes, "let the agent ask for more", kept inside one server, is a fair way to put it.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
That's the version I'd actually trust — a lookup that happens up front, on purpose, costs less than one that happens after a failed call, and the worked examples mean the agent isn't guessing the payload shape mid-edit. Still an extra call either way, just one you can budget for instead of one that surprises you. 'Let the agent ask for more' is a fair way to put it, as long as it's one server asking, not five.
I'd compare them on completed editing tasks too: full schemas versus lazy lookup, same tasks and model. Then count successful edits, total tokens and latency, including retries. An extra lookup could pay for itself if it avoids wrong calls; the size of the tool list alone doesn't tell us that.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
That's the test that actually settles it — the list size never will. Same tasks, same model, full schemas against lazy lookup, and count what matters: edits that land, tokens, time with the retries in. I'll run that one.
Developer Advocate exploring APIs, AI agents, developer tools, and open-source projects. Passionate about learning, building, and connecting with developers.
Location
Amsterdam,Netherlands
Education
B.E. Computer Science, Chandigarh University (2022–2026)
That does make a noticeable difference between 122 tokens and 262 tokens for each tool. I thought it was an interesting point you made about descriptions increasing cost. Have you tried loading in tool definitions only when needed by an agent?
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
Not yet — everything in this one was loaded up front, that's what the numbers are measuring. The lazy version is the next run, and there's a thread just below working out how to measure it fairly.
Completed tasks is the test that settles it. The schema goes back over the wire on every turn, so its cost is its size times the number of turns run. I measure that on THOL: cost per solved task, checked by a verifier. (I built Tokenade.)
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
Fair — 24,522 is the per-message number, and the bill that matters is that times however many turns a task actually takes. I haven't measured cost per solved task yet, that's the hole in this piece, so I'm going to run that one — same tasks, trimmed tool list, see what the per-task bill actually looks like. What kind of turns-per-task are you seeing on THOL with a list this size?
@aheadkit Turns set the bill, so THOL ranks tools by cost per solved task. Every run records its turn count next to its cost, and the raw per-run results are published with the leaderboard. I'd rather point you there than quote a turns figure for a list this size.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
Fair enough, the leaderboard's the number then. Do the per-run rows break out by list size, or does a 200-tool run sit in the same ranking as a 20-tool one?
We're a place where coders share, stay up-to-date and grow their careers.
A data point from a real server, counted your way (cl100k, compact JSON). Disclosure: I maintain CutVoke, an open-source video editor that agents drive over MCP.
Its server lists 52 tools, which come to 7,958 tokens, about 153 per tool. That's heavier than your synthetic ones, partly because the descriptions are in Chinese. Your whitespace point holds too: the same 52 pretty-printed are 12,567.
The bigger number is what isn't on the menu. There are 110 edit commands, and their payload schemas together come to 35,965 tokens. Instead of 110 more tools they all go through one
command_apply(type, payload), and the agent fetches the schema for the one command it needs withcommands_list(type=...)oreditor_help(topic="command", command=...)right before using it. Even the name-only catalog (5,924 tokens) is fetched on demand rather than listed.The trade-off: an unfamiliar command costs an extra round trip, and the model only sees a schema after it asks. An unknown command name comes back with an error pointing to
commands_list. So your "let the agent ask for more" also works inside a single server, not only by splitting servers.github.com/frankShan001/CutVoke
This is the number I was missing — 153 a tool on a real server sits right between my lean synthetic ones and the filesystem server's 263, and the Chinese descriptions explain part of it. But the 35,965 hiding behind the lazy fetch is the part that got me: the menu looks cheap because the real schemas only show up when the agent asks. That trade-off's real too, an extra round trip every time the model meets a command it doesn't know. You basically built the version of 'let the agent ask for more' that lives inside one server instead of across several.
Yes, that round trip is the part I keep weighing. Two things in the server try to make it land at the right moment: the MCP instructions tell the agent to call
editor_help(topic='command', ...)orcommands_listbefore an unfamiliar edit, so the lookup happens up front rather than after a failed call, andeditor_help(topic='recipes')has worked examples with the payload shapes for common edits. It doesn't remove the extra call, it just makes it predictable. And yes, "let the agent ask for more", kept inside one server, is a fair way to put it.That's the version I'd actually trust — a lookup that happens up front, on purpose, costs less than one that happens after a failed call, and the worked examples mean the agent isn't guessing the payload shape mid-edit. Still an extra call either way, just one you can budget for instead of one that surprises you. 'Let the agent ask for more' is a fair way to put it, as long as it's one server asking, not five.
I'd compare them on completed editing tasks too: full schemas versus lazy lookup, same tasks and model. Then count successful edits, total tokens and latency, including retries. An extra lookup could pay for itself if it avoids wrong calls; the size of the tool list alone doesn't tell us that.
That's the test that actually settles it — the list size never will. Same tasks, same model, full schemas against lazy lookup, and count what matters: edits that land, tokens, time with the retries in. I'll run that one.
That does make a noticeable difference between 122 tokens and 262 tokens for each tool. I thought it was an interesting point you made about descriptions increasing cost. Have you tried loading in tool definitions only when needed by an agent?
Not yet — everything in this one was loaded up front, that's what the numbers are measuring. The lazy version is the next run, and there's a thread just below working out how to measure it fairly.
Completed tasks is the test that settles it. The schema goes back over the wire on every turn, so its cost is its size times the number of turns run. I measure that on THOL: cost per solved task, checked by a verifier. (I built Tokenade.)
Fair — 24,522 is the per-message number, and the bill that matters is that times however many turns a task actually takes. I haven't measured cost per solved task yet, that's the hole in this piece, so I'm going to run that one — same tasks, trimmed tool list, see what the per-task bill actually looks like. What kind of turns-per-task are you seeing on THOL with a list this size?
@aheadkit Turns set the bill, so THOL ranks tools by cost per solved task. Every run records its turn count next to its cost, and the raw per-run results are published with the leaderboard. I'd rather point you there than quote a turns figure for a list this size.
Fair enough, the leaderboard's the number then. Do the per-run rows break out by list size, or does a 200-tool run sit in the same ranking as a 20-tool one?