- Book: AI Agents Pocket Guide: Patterns for Building Autonomous Systems with LLMs
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
A team I talked to last quarter shipped their first agent with 6 tools and 84% eval pass rate. Three months later they had 38 tools and 67%. They blamed the model. They tried a bigger model. They rewrote the system prompt twice. The accuracy stayed flat.
The catalog was the bug.
The catalog grew, the eval suite shrunk
The pattern is predictable. Quarter one ships with search_docs, lookup_customer, create_ticket, send_email, get_calendar, escalate_to_human. Six tools. Clean. Then PM wants the agent to "also handle refunds": process_refund, get_order, get_payment. Then sales wants quotes: lookup_price, apply_discount, generate_quote. Then someone notices search_docs is too broad and splits it into search_kb, search_runbooks, search_policies. Then the integrations team adds eight Salesforce wrappers because the model "should just have access to everything."
You wake up at 38 tools. Your eval set, which you never grew alongside the catalog, says you went from 84% to 67%. The CEO asks why the new model isn't helping.
The model isn't the problem. Tool selection is.
Why this happens (it's the softmax)
When an agent picks a tool, the model is producing a probability distribution over tool names and descriptions. The relevant pieces (names, descriptions, parameter schemas) all live in the same context window, competing for attention. Tool selection ends up looking a lot like a softmax over name similarity to the user query.
Two things degrade as the catalog grows.
The first is probability dilution. With 6 tools, the right one might get 70% of the mass and the runner-up 12%. With 40, the right one gets 18% and three near-duplicates get 12-14% each. The model still ranks the right one first on easy queries. On the queries where the user phrasing is ambiguous or partially matches multiple tools, it doesn't.
The second is description overload. Each tool definition is typically 80-200 tokens with parameter schema. Forty tools is 5-8k tokens of tool docs the model has to attend to before generating a single character. Context window math is fine. Attention isn't free. The model's effective "tool comprehension" window is much smaller than the raw token budget.
There's also a quieter failure: the model picks a plausible but wrong tool and proceeds with junk arguments because the description didn't disambiguate hard enough. You see this as Tool returned 0 results followed by a confident "I couldn't find anything in our system", when the agent actually searched the wrong index.
Find your own curve in 60 lines
Don't trust my numbers. Find yours. Here's an A/B script that takes a fixed eval set, runs it against your agent at catalog sizes 5/10/20/30/40/50, and prints the S-curve.
import os
import random
import json
from dataclasses import dataclass
from anthropic import Anthropic
client = Anthropic()
random.seed(42) # reproducible decoy sampling
@dataclass
class EvalTask:
query: str
expected_tool: str # ground-truth tool name
# Your real catalog. Replace with yours.
CATALOG = [
{"name": "search_kb", "description": "Search internal knowledge base articles", "input_schema": {"type": "object", "properties": {"query": {"type": "string"}}}},
{"name": "lookup_customer", "description": "Look up customer by email or ID", "input_schema": {"type": "object", "properties": {"identifier": {"type": "string"}}}},
# ... 48 more real tools from your prod catalog
]
# 100 hand-labeled eval tasks. Each query has one correct tool.
EVAL: list[EvalTask] = [
EvalTask("how do I reset my password", "search_kb"),
EvalTask("find the order for jane@acme.io", "lookup_customer"),
# ... 98 more
]
def run_one(task: EvalTask, tools: list[dict]) -> bool:
"""Single API call. Returns True if model picked the right tool."""
resp = client.messages.create(
model="claude-haiku-4-5",
max_tokens=512,
tools=tools,
tool_choice={"type": "any"}, # force a tool pick
messages=[{"role": "user", "content": task.query}],
)
for block in resp.content:
if block.type == "tool_use":
return block.name == task.expected_tool
return False
def sample_catalog(size: int, must_include: set[str]) -> list[dict]:
"""Build a catalog of `size` tools that includes the ground-truth
tools needed for the eval set, padded with random decoys."""
needed = [t for t in CATALOG if t["name"] in must_include]
pool = [t for t in CATALOG if t["name"] not in must_include]
extras = random.sample(pool, max(0, size - len(needed)))
cat = needed + extras
random.shuffle(cat) # don't let position bias help us
return cat[:size]
def sweep(sizes: list[int]) -> dict[int, float]:
needed = {t.expected_tool for t in EVAL}
if any(t["name"] not in {x["name"] for x in CATALOG} for t in EVAL):
raise ValueError("eval references tool not in catalog")
results = {}
for size in sizes:
if size < len(needed):
continue # can't run, ground truth wouldn't fit
catalog = sample_catalog(size, needed)
correct = sum(run_one(t, catalog) for t in EVAL)
acc = correct / len(EVAL)
print(f"size={size:>3} accuracy={acc:.2%} ({correct}/{len(EVAL)})")
results[size] = acc
return results
if __name__ == "__main__":
sweep([5, 10, 20, 30, 40, 50])
A few notes before you run this and accuse me of cherry-picking.
Pick Haiku for the sweep, not Sonnet. Two reasons. One, you'll burn through 600 calls per sweep and want to do it twice. Two, the smaller model surfaces the selection error earlier and louder. If Haiku flatlines at 50 you'll see the shape clearly. Sonnet smooths it and you'll think you don't have a problem until prod traffic finds the soft spot. Run Sonnet last to confirm.
The seeded random.sample is the gotcha most people miss. If you re-roll decoys per size, the noise drowns the signal. Same seed, same decoys grow into the catalog as size increases. That's the experiment.
What you typically see when you actually run this on a real catalog:
size= 5 accuracy=92.00%
size= 10 accuracy=89.00%
size= 20 accuracy=83.00%
size= 30 accuracy=71.00%
size= 40 accuracy=63.00%
size= 50 accuracy=58.00%
The shape is an S, not a line. Flat top from 5-15, sharp elbow somewhere in the 20-35 band, long tail of slow degradation past 40. Where exactly the elbow lands depends on your descriptions, your query distribution, and your model. The team I mentioned earlier found theirs at 28. Another team I know found theirs at 22 because they had a lot of near-duplicate Salesforce wrappers.
The tool-budget pattern
Once you know your elbow, you have a number: the per-turn tool budget. Pick something below it. 10-15 is a useful default. The question is how to choose which 10-15 to expose per turn without changing your agent framework.
Three options, ranked by latency and complexity.
Option A: cheap classifier. Run the user query through a small embedding model or a Haiku call that returns a category. Map category to a pre-baked tool set.
CATEGORIES = {
"billing": ["lookup_customer", "get_invoice", "process_refund", "get_payment", ...],
"support": ["search_kb", "create_ticket", "escalate_to_human", ...],
"sales": ["lookup_price", "apply_discount", "generate_quote", ...],
# ... etc
}
def route(query: str) -> list[dict]:
cat = classify(query) # tiny Haiku call or local classifier
names = CATEGORIES[cat] + GLOBAL_TOOLS # always include search_kb etc.
return [t for t in CATALOG if t["name"] in names]
Adds 200-400ms. Works well when intents are clean. Falls apart when a single user message touches two categories ("can you refund my last order and email me a confirmation"). Mitigation: let the classifier return multiple categories and union the sets, capped at your budget.
Option B: embedding similarity. Pre-embed every tool's description. Embed the user query. Take top-K by cosine similarity.
import numpy as np
# Done once at startup.
TOOL_VECS = {
t["name"]: embed(t["name"] + ": " + t["description"])
for t in CATALOG
}
def route(query: str, k: int = 12) -> list[dict]:
q = embed(query)
scored = [
(name, float(np.dot(q, vec)))
for name, vec in TOOL_VECS.items()
]
top = sorted(scored, key=lambda x: -x[1])[:k]
keep = {name for name, _ in top}
return [t for t in CATALOG if t["name"] in keep]
50-150ms with a local model, 200-300ms with a hosted one. Better than the classifier on cross-cutting queries. Worse when tool names lie: lookup_customer that actually does customer + order + payment is not embedded honestly.
Option C: deterministic capability map. Hand-write rules that map query patterns or user roles to tool subsets. Boring. Fast. Predictable. Works great when you have strong domain structure (e.g., your agent serves three product lines and tool relevance is mostly product-line-bound).
def route(query: str, user: User) -> list[dict]:
names = set(GLOBAL_TOOLS)
if user.has_role("billing"):
names |= BILLING_TOOLS
if "refund" in query.lower():
names |= REFUND_TOOLS
# ... etc
return [t for t in CATALOG if t["name"] in names]
The best fit for compliance-heavy environments where you need to prove a given user couldn't have triggered a given tool. The worst fit for open-ended assistants.
Most production systems end up combining (B) and (C): role-based gating with embedding similarity ranking inside the role's allowed subset.
Naming and descriptions matter more at scale
At 6 tools, search is fine. At 30, search is an attack on your own agent.
The discipline at 20+ tools: every tool name should be unique on the first 8 characters and every description should answer "what does this NOT do?"
# Bad
search: "Search internal docs"
lookup: "Look up information about a customer"
find: "Find records"
# Better
search_kb: "Search the public-facing help center articles.
Use for product questions, not customer data."
lookup_customer: "Get one customer record by email or customer ID.
Does not include orders or payments. Use
get_customer_orders for those."
search_internal_runbook: "Search engineer-only operational runbooks.
Use only when user is an internal staff member."
The disambiguation language ("not", "only when", "use X instead for Y") is what cuts selection error at scale. Models pick the wrong tool because two descriptions sound interchangeable, not because they don't know what the right tool does.
What to cut, what to merge
Run this once a month against your traces:
- Any tool with <2% invocation share that isn't a safety-critical path (escalate, refund, delete): cut it. The model isn't using it, the catalog cost is real.
- Any pair of tools where >40% of invocations are followed by the other within two turns: candidate for merging into one tool with a
modeparameter. - Any single-API-call wrapper that doesn't add validation, side-effect tracking, or argument coercion: collapse it into the parent tool.
- Any tool whose last 50 invocations all took the same default argument: make it the default, drop the parameter.
The team I mentioned cut from 38 to 19 in a week using just these four rules and got from 67% back to 81%. They didn't lose any user-visible capability. They lost confused selections.
When 30+ is fine (hierarchical dispatcher, honestly)
If your domain genuinely has hundreds of operations (think a general-purpose enterprise assistant with read access to seven internal systems), there's a pattern that works: a hierarchical dispatcher. The top-level agent sees ~8 broad tools (query_crm, query_billing, query_calendar, dispatch_action, ...). Each top-level tool is itself an agent with its own catalog of 10-15 operations. Two-level tree, branching factor of about 10.
This works. I'd be lying if I said it works cleanly.
The honest scope-out:
- Latency doubles. Two LLM calls in series per turn, minimum.
- Traceability gets worse. You now have two reasoning steps to debug instead of one. Your trace viewer needs to render the nesting properly or your on-call team will hate you.
- Cost goes up unless the inner agents run on a cheaper model, which is itself a routing decision you have to defend.
- Failure modes multiply. Top-level picks the wrong subordinate (~5% of the time even with careful prompting), and now you're debugging a wrong-tool problem two layers deep.
- The 20-tool elbow shifts down for the dispatcher itself. If your top-level has 12 dispatcher tools and you add three more, that level starts degrading too.
Use the hierarchical dispatcher when you genuinely have >50 operations and you've measured that single-level routing tops out below your accuracy bar. Don't reach for it because the catalog feels big. Most catalogs feel big at 25 and aren't.
The first thing to try is pruning. The second is budgeting. Dispatcher is third.
The shortest possible version
Your catalog has a curve. The model gets dumber as you add tools. The elbow is usually between 20 and 35. Measure yours. Budget below it. Cut tools nobody uses, rewrite descriptions for disambiguation, and only go hierarchical when you've proven the single level can't hold.
Where does your catalog sit today, and have you measured the elbow or are you guessing? Drop your tool count and accuracy delta in the comments. I'm curious how steep the cliff is in other teams' production setups.
If this was useful
Tool-catalog management is one of the operational disciplines that turns demo agents into systems people actually depend on. The AI Agents Pocket Guide covers the curve in more depth, plus the chapters on tool budgeting, dispatcher topologies, and the eval patterns you need before you start cutting tools. If your catalog is past 25 and your accuracy is wobbly, that's the book.
Top comments (0)