Raw accuracy fell 25 points at 50 tools, but a chance-corrected read tells a very different story.
The request was simple: "Email Priya the Q3 numbers right now, subject 'Q3 Numbers'." The agent had a tool called send_email, correctly named, correctly described, sitting right there in its list of available tools. It did not call it. It called find_contact(name="Priya") instead, and stopped.
Nothing about the request changed. The model didn't change. The only thing that changed between a correct answer and this one was how many other tools were sitting in the same menu: 5 the first time, 50 the second. I ran the same 20 requests against the same model four times, once per tool-menu size, to find out how much of that gap is the model actually getting worse at choosing, and how much is just more wrong options to pick from. Those turned out to be two different questions with two different answers.
What was actually run: 20 fixed scenarios, each with one correct target tool out of a 50-tool bank spanning email, calendar, messaging, weather, currency, maps, files, and a dozen other everyday domains. Each scenario also named 2-4 real near-neighbor decoys (tools a reasonable person would also consider) so even the smallest menu had genuine confusion built in, not just an easy target and random padding. Four tool-count conditions: 5, 10, 25, 50. Model: gemini-3.5-flash-lite, free tier, real function-calling (not a scripted keyword match), temperature 0, single trial per (scenario, tool-count) pair, 80 calls total. Zero cost. Full raw data, scoring code, and the chart below are pinned at commit d078df3.
A real mid-run constraint, not hidden: the first attempt used a different model (gemini-3.6-flash) and hit that model's free-tier daily cap at 20 requests before finishing a single condition. Switched to a different model with its own quota bucket for the full run. Logged in configs/run.json because an experiment's friction is part of its record, not something to edit out.
What the 80 calls actually showed:
- Raw accuracy: 80% (5 tools) → 80% (10) → 75% (25) → 55% (50). The real cliff is between 25 and 50, not a smooth slope.
- Chance-corrected, the model got relatively better as the menu grew: 4x better than random guessing at 5 tools, 27.5x better at 50. Both numbers are true at once.
- At small menus, wrong answers were mostly real near-neighbor mix-ups (SMS vs. WhatsApp). At 50 tools, 8 of 9 wrong answers called a tool with no real connection to the request.
- One tool,
find_contact, absorbed 9 of those 14 unrelated wrong calls across every condition, including the exact case above where the correct tool was present and correctly named. - 3 of the 20 scenarios were wrong at every menu size, including 5 tools. Those aren't a tool-count effect, and treating them as one would overstate the finding.
Tool-selection accuracy by menu size, with the pure-chance baseline (1/n) for comparison. Image by author.
The question going in was simple: does giving an agent more tools make it worse at picking the right one? That's testable, and it's also already been tested. RAG-MCP reported tool-selection accuracy falling from 43.13% to 13.62% as a tool index grew, more than tripled back by retrieval-based filtering. ToolChoiceConfusion found the same shape across 2,448 runs and four model backends, and named the exact failure this experiment went looking for: tools that are "plausible but not appropriate." What neither paper's headline number answers on its own is whether a 25-point drop in raw accuracy means the model's judgment actually degraded, or whether it's an artifact of a shrinking chance of a correct guess by luck alone.
Method, in enough detail to rerun it. generate_bank.py builds a fixed 50-tool bank across 15 domains (no randomness: this is hand-authored so every tool name and description is stable across runs) and 20 scenarios, each with a target tool and named near-neighbors. For each (scenario, tool-count) pair, run_experiment.py builds a menu containing the target, the scenario's own near-neighbors, and enough randomly sampled filler to reach the target size, seeded for reproducibility, then shuffles the menu order so position can't leak the answer. The call itself goes through Google's generateContent REST endpoint with functionDeclarations as the tool schema and toolConfig.functionCallingConfig.mode forced to ANY, so the model has to call something rather than reply with text. A result counts as correct only if the returned function name exactly matches the scenario's target.
def build_menu(scenario, all_tools_by_name, tool_count, rng):
"""Correct tool + near-neighbors first, then random fill to tool_count."""
target = scenario["target"]
chosen = [target] + [n for n in scenario["near_neighbors"] if n != target]
remaining = [name for name in all_tools_by_name if name not in chosen]
rng.shuffle(remaining)
while len(chosen) < tool_count and remaining:
chosen.append(remaining.pop())
chosen = chosen[:tool_count]
rng.shuffle(chosen) # position must not leak the answer
return [all_tools_by_name[name] for name in chosen]
Results. 16/20 (80%) correct at 5 tools. 16/20 (80%) at 10. 15/20 (75%) at 25. 11/20 (55%) at 50. That last number is the one a headline would normally stop at, and it would be telling the truth: a 25-point fall, concentrated almost entirely in the jump from 25 to 50 tools, is a real and practically important failure mode. But raw accuracy and chance accuracy aren't the same denominator. At 5 tools, guessing at random gets you 20% right by luck alone, so 80% is 4 times better than chance. At 50 tools, random guessing gets you 2%, so 55% is 27.5 times better than chance, the best relative performance in the whole experiment. The model's accuracy relative to pure chance climbed every single time the menu grew, even as its raw accuracy fell. Reporting only the raw number isn't wrong, but it answers a different question than "is the model's judgment actually getting worse," and conflating the two overstates the finding in exactly the direction that makes for a scarier headline.
Where it actually broke, and what changed as the menu grew. With only 5 tools on the menu, 3 of the 4 wrong answers were real near-neighbor confusions: asked to text someone, the model sent a WhatsApp message instead of an SMS, a mistake a careful human could plausibly make too, since both tools do roughly the same job. By 50 tools, that pattern flips: 8 of 9 wrong answers called a tool with no real relationship to the request at all, and one tool in particular, find_contact, accounted for 9 of the 14 unrelated wrong calls across every condition combined. The clearest single example is the one that opened this piece: send_email was present, correctly named, and correctly described in the 50-tool menu, and the model still reached for find_contact on seeing a person's name in the request. That specific failure never happened at 5, 10, or 25 tools for the same request, same model, same temperature. Something about the sheer size of the option set pulled the model toward a generically-plausible action instead of the one it had already gotten right three times at smaller sizes.
The honest caveat: 3 of the 20 scenarios (asking it to text someone, book a call, and check availability) were wrong at every single tool count, including the 5-tool floor. Those are weaknesses in how those three requests were worded against their near-neighbor tools, not a tool-count effect, and folding them into the headline number would be cherry-picking the story in the convenient direction. The actually isolated tool-count effect is the 5 scenarios (a quarter of the set) that were answered correctly at low tool counts and became wrong specifically and only as the menu grew, with nothing else in the setup changing.
What this means if you're building one of these. If your agent's tool count is staying under 10-15, this experiment doesn't predict much new reliability risk from scaling it a little further, and the near-neighbor confusions you'll hit there are the ordinary cost of giving two tools similar jobs; disambiguating their descriptions is the fix, not routing them away. Past 25-30 tools, Cursor's own hard 40-tool cap, and an independent measurement of Claude Code's quality degrading past 50 tools both suggest real products are already drawing this exact line for a reason. The fix the field has converged on, and the one this experiment's own numbers support, is retrieval-based tool search: show the model a short, relevant shortlist per request instead of the full menu every time. MCP's own 2026 spec changes added progressive tool discovery for exactly this reason. What this experiment adds on top of that existing advice: watch for a single generically-named tool becoming a disproportionate failure sink as your toolset grows, the way find_contact did here. That's a cheap thing to check with a confusion matrix over your own logs, and a much narrower fix (rename or re-scope one tool) than "add a retrieval layer."
Reproducibility. Everything above, the tool bank, the 20 scenarios, the exact menus used for all 80 calls, the raw responses, the chart code, and both full failure transcripts, is public at github.com/priyansh18/tool-count-confusion, pinned to commit d078df3. methodology.md has the full design rationale and stated limitations (20 scenarios is a toy sample, single-trial at temperature 0, no argument-correctness scoring); analysis.md has the complete breakdown behind every number above.
If you're building an agent with a growing tool count right now: would you rather catch this with a retrieval layer that filters the menu before the model ever sees it, or with per-tool telemetry that flags a specific tool, like find_contact here, absorbing failures that belong to other tools?
Disclosure: this article was drafted with AI assistance (Claude), including the experiment code, from a real, live-run experiment and source-checked claims; the model, its tools, and its config are all public in the linked repo for independent verification.
If you want a deeper, structured path through this material rather than one article at a time, I've been building a free roadmap at notes.priyanshsinghal.com as I study it myself, and I take on 1:1 sessions for people working through the same material at topmate.io/priyansh_singhal10.
Top comments (2)
From the Auten team (we ship a computer-use MCP server, so our tool list is long too). Did find_contact soak up those calls because its description overlapped the others, or because of where it sat in the list? I'd start with your per-tool telemetry: a cheap first fix is one "don't use this for X" line in each overlapping description, and the telemetry tells you whether it worked before you add a retrieval layer.
The find_contact detail is the one that should worry tool builders most — 9 of 14 unrelated wrong calls landing on the same decoy means the failure isn't random confusion, it's a magnet. And the chance-corrected read is the honest headline: the model got relatively better as the menu grew, raw accuracy just couldn't outrun the denominator. Do you think the fix is retrieval-based filtering before the model ever sees the menu, or does that just move the selection problem into the retriever?