Your tool is registered. It appears in tools/list. The model never calls it, or calls a different one instead.
There are six reasons this happens....
For further actions, you may consider blocking this person and/or reporting abuse
The strongest description is an executable promise: the input it accepts, the state it reads or changes, the output shape, and the conditions that make it the right choice over its siblings. I’d test tool selection with contrastive prompts that make neighbouring tools plausible, then use the mistaken selection as a regression case for the schema and descriptions.
"Executable promise" is a better spec than anything in the post, and the fourth item is the one nobody writes. Input, state, output shape — those show up often enough. The conditions that make this the right choice over its siblings almost never do, because writing it requires holding the whole tool list in your head at once, and people write descriptions one tool at a time in the file where they registered it.
Which is probably the mechanism behind the 17.7% rather than carelessness. Every description in that Gmail example is correct about its own tool. None of them was written with the others open.
Contrastive prompts as the test is the part I want to steal. My static check finds pairs a model might confuse; it can't tell which confusions are real. Deliberately writing a prompt that makes two neighbouring tools plausible and seeing which one gets picked converts a suspicion into a fact, and it needs no traffic — you can do it before shipping.
The regression case is the bit that makes it durable. Each mistaken selection becomes a fixture: prompt, expected tool, actual tool. Then a description rewrite is testable rather than hopeful, and the next person who "improves" the wording finds out immediately if they've broken the distinction. That's the thing missing from every schema-linting approach including mine — a linter tells you a description looks risky, a regression suite tells you whether the fix worked.
It also generalises past MCP. Any tool-using agent has this problem, and the artefact is just a labelled set of intent-to-tool pairs. Which is the thing a schema analysis can never produce on its own, because it's the ground truth I'd be validating against.
Contrastive tests become much more useful when the fixture includes the nearby tools the model should reject, not only the intended one. I would version those candidate sets and track selection failures separately from execution failures, so a change in wording cannot hide a routing regression behind a passing unit test.
Versioning the candidate set is the part that makes the fixture trustworthy, and I'd have left it out.
Without it the test is unfalsifiable in a quiet way. A probe passes, you assume the rewrite worked, and you have no idea whether the competitor that was absorbing the request is even still in the list. Someone deleted it, or renamed it, or it moved behind a feature flag — and the regression suite reports green for the wrong reason. The fixture has to record what the model was choosing between, not just what it chose.
That generalises past your own server too. The candidate set a model actually sees is the union of every connected server, so a pair that's stable in isolation can split once a neighbour with overlapping input shows up. Versioning the set is the only way a split is traceable to a cause rather than just observed.
Separating selection failures from execution failures is the other half, and it's a distinction I'd been making in the metrics and not in the testing. Wrong tool chosen is a schema and description problem. Right tool chosen, arguments malformed, is a parameter problem. Right tool, right arguments, wrong answer is your handler. Three different teams' worth of work, and a single pass/fail collapses them into one number that moves for reasons nobody can attribute.
The failure mode you're describing — a wording change hiding a routing regression behind a passing unit test — is the specific thing that makes selection worth isolating. Execution tests are the ones people already have, so they're the ones that stay green and provide false comfort.
That separation also suggests a useful test-report shape: record the candidate-set version, selected tool, argument-validation result, and handler assertion as distinct fields for each probe. A failure can then point to routing, parameter construction, or execution without asking a later reader to infer the layer from a single red result.
Four fields rather than one verdict is the right shape, and the reason it matters is less about the person reading the failure than about what happens to a suite over time.
A single red result gets triaged by whoever is on duty, who infers a layer from the error text and fixes at that layer. Guess wrong and you've made a change that doesn't address the cause while the test goes green — the failure moved rather than resolved. Four recorded fields remove the inference step entirely, so the triage can't be wrong about which layer it was.
One addition I'd make now I'm looking at the four: record the candidate set itself, not only its version. A version identifier is enough to detect that the set changed, and not enough to say what changed, so a historical failure becomes unreadable once the version has moved on twice. Storage is trivial — it's a list of tool names — and it means an old report is still interpretable.
The argument-validation field is also doing something subtler than it looks. A probe that routes correctly and then fails validation isn't a routing success with an execution problem attached. It's evidence the description and the schema disagree: the description persuaded the model this was the right tool and the schema then rejected what it sent. That's one coherent failure spanning two fields, and it only becomes visible because they're recorded separately.
Which argues for the handler assertion being optional, incidentally. Most of what I'd want from a probe suite is answered before the handler runs at all, and requiring an execution assertion makes probes expensive enough that fewer get written.
Cause 4 is the one that took me longest to accept was the actual problem. Under fifteen tools you can mostly brute-force it and lean on description quality. Past thirty I've had to go to dynamic registration: a lightweight classifier reads the opening message and loads only the tools plausibly relevant to that intent. The catch is the classifier can't use the full tool set to decide what to include, so you've quietly turned a routing problem into a retrieval problem.
"You've turned a routing problem into a retrieval problem" is the sentence I'll be repeating, because it names the cost precisely. The routing failure is visible — wrong tool called, and you can see which. The retrieval failure is silent: the right tool was never a candidate, nothing was called, and nothing in your logs distinguishes that from nobody having asked.
Which makes cause 1 nastier under dynamic registration rather than better. If your classifier reads the opening message and your tool descriptions are what it reads, two tools a model can't tell apart are also two tools a classifier can't tell apart — you've moved the same ambiguity one layer earlier, where it's harder to observe. The description quality problem doesn't go away, it just stops producing a wrong call you can point at.
It also splits my measurement in a way I hadn't accounted for. Schema cost per connection assumes the whole list ships every time, which for you it doesn't — your median payload is whatever the classifier selected. And dead tools becomes ambiguous: never called might mean never selected by the model, or never loaded by the classifier, and those have completely different fixes.
The thing I'd want to know, if you ever get a way to measure it: how often the classifier excludes a tool that would have been the right answer. That's the retrieval failure rate, and it's the number that tells you whether dynamic registration is net positive or just relocating the problem. I don't know how you'd get it without a ground truth set of intent-to-tool pairs.
Fifteen and thirty as the two thresholds matches the corpus, incidentally — collisions run at 4.3% up to fifteen tools and 16.3% by sixty.
There is one more failure class between schema and reachability: the tool is callable but policy-ineligible in the current context. A model can correctly avoid a well-described tool because the user did not authorize its side effect, the required identity is unverified, or the destination is ambiguous. Instrumenting only selection rate would mislabel that restraint as poor discoverability. I would log candidate, eligible, selected, attempted, and succeeded as separate stages.
That's a real gap and it's worse than a missing category — it's a sign error. Every cause in the post is something to fix. Correct restraint measured the same way reads as a problem and produces advice that makes the server less safe: rewrite the description, make it more findable, get the model to call the thing it was right to avoid.
The five stages are the fix, and the gaps between them carry different meanings. candidate → eligible is a policy filter and no schema change touches it. eligible → selected is where everything in the post lives. selected → attempted catches arguments the model couldn't fill. attempted → succeeded is the handler.
Collapsing those into one selection rate makes them indistinguishable, which is what I'm currently doing.
The honest problem is that eligibility is invisible from where I sit. An SDK inside the server sees the call arrive or not arrive; it can't see the model declining because the user never authorised a side effect, or because the identity behind the request is unverified. That decision happens in the client, and the protocol reports nothing back.
Which means candidate → eligible is either author-declared — the tool itself knows it requires an authorised identity or a disambiguated destination, and can say so — or it stays unobservable and I should say that rather than folding it into selection.
Declared is probably right, and it has a useful side effect: a tool that declares a policy precondition can be excluded from the discoverability numbers entirely, so a well-designed dangerous tool stops being penalised for not getting called.
Adding this. It's the first thing anyone has raised where the existing metric doesn't just miss something but actively points the wrong way.