You can read your own tool descriptions all day and not know whether a model can tell them apart. I spent a few weeks measuring 82,549 public MCP tool descriptions, and the most useful thing I learned is that the measurement has a ceiling.
Here are two tools from a real server:
score_resume "Score a resume for ATS compatibility."
analyze_job_description "Extract what a job posting actually screens on."
They share no content word. Not one, even after stemming. Every lexical metric I have scores them as perfectly distinct.
Their author says they're confusable, and he's right — because "check my resume for this job" names both objects at once. The collision doesn't live between the two descriptions. It lives between a sentence and a set of tools, and nothing computed from descriptions alone will ever see it.
So here's the test that does. It takes an afternoon, needs no traffic, and produces evidence you can put in a pull request.
The method came out of a comment thread on a previous post — credit where it's due, I'd been circling something much more expensive.
The shape of it
For each pair of tools you suspect might be confusable, write two sentences:
- A control — a request that should route to exactly one of them, unambiguously.
- A probe — a request that could fairly route to either.
Run each through a model with the full tool list, at fixed temperature, a handful of times. Record which tool gets picked.
That's the whole method. The interesting part is reading the results.
Reading the control
The control runs first, and it's a gate rather than a finding.
If the control doesn't route correctly, stop — you don't have an ambiguity problem between this pair. You have one description that's broken on its own terms, and that's a different fix. Rewriting it in relation to its sibling would be solving the wrong problem.
Most of the time the control passes and you move on. When it fails, it's usually worth more than anything the probe would have told you.
Reading the probe
This is where the instinct misleads.
If the probe routes to the same tool every time, that's a pass, even though the sentence was genuinely ambiguous to you. It means the model has found a distinction you can't see in the words. Maybe it's the parameter names. Maybe it's the tool name. Maybe it's something about the phrasing you didn't consciously encode. Doesn't matter — it works.
My instinct would have been to flag any non-uniform distribution as a collision. That's wrong, and acting on it would mean rewriting descriptions that are already doing their job.
If the probe splits, that's a real collision, and you now have the sentence that proves it. Not a suspicion. Not a metric above a threshold. An actual user request that lands in two different places depending on the roll.
That distinction — stable versus split — is the whole reason this is worth running. And a handful of runs at fixed temperature is enough to tell them apart, which is a far smaller budget than I'd assumed before someone spelled it out.
Picking the pairs
You can't do this pairwise on a large server. 191 tools is 18,145 pairs, and at even a handful of model calls each, that's not an afternoon.
This is where static analysis earns its place. It stops being the answer and becomes the filter that produces candidates.
Rank pairs by input overlap — do these two tools act on the same object? Compare the nouns in their descriptions plus their parameter names. Tools that share their input are the ones where a single sentence can route to either, which is exactly the failure the probe tests for.
Then gate that on output distinguishability. If two tools take the same input but their descriptions clearly state different returns — "flat details for many records" versus "a bounded evidence bundle" — that's good design, not a problem. Don't spend probes on it.
Rank, take the top fifty, probe those. The static metric was never going to find the collision; it's very good at telling you where to look.
What to do with a split
Three outcomes, and they're genuinely different kinds of work.
Rewrite for contrast. The cheapest fix and it changes no behaviour. The rule that works: every tool in a cluster shares its input by definition — that's what makes them a cluster — so describing the input describes what they have in common. Name the output instead. "Search reports" describes ten tools. "Search reports by client, including archived ones, returning IDs and titles only" describes one.
State the trigger condition. Call this one before acting, that one after a decision is made. No lexical measure sees this and it's frequently the real distinction. It's also the part a description template flattens, which is why collision rates climb sharply past about thirty tools — that's where people stop writing descriptions individually.
Merge into one tool with a mode parameter. Sometimes the split isn't telling you your wording is bad. It's telling you the sentence names an intent your tools have partitioned and the user hasn't. "Check my resume for this job" is not a failure of either description — it's a request that legitimately spans both. If two tools can ever be correct for the same sentence, you may not have two tools. You may have one tool with a parameter.
That last one is the conclusion I've now reached from two completely different directions: as a design rule about interfaces, and empirically from a probe that splits. The second version is the one you can put in a pull request without arguing, because it comes with the sentence attached.
Keep the failures
Every split is a regression case: the prompt, the expected tool, the tool actually picked.
Store them. Then a description rewrite stops being hopeful and becomes testable — you re-run the probe and see whether the split closed. And the next person who "improves" the wording six months from now finds out immediately if they've broken the distinction.
This is the thing missing from every schema-linting approach, mine included. A linter tells you a description looks risky. A regression suite tells you whether the fix worked.
The limits
Fixed temperature and a handful of runs gives you a signal, not a rate. If you want "this pair splits 60/40", that needs many more runs and it's probably not worth the budget — the useful information is binary.
Different models may split differently, which is itself worth knowing. If a pair is stable in one client and splits in another, the problem is more specific than your descriptions.
And a probe that doesn't split tells you about that sentence, not about every sentence. Someone will eventually phrase something you didn't think of. That's an argument for keeping the regression suite and adding to it, not for distrusting the method.
Why this is the interesting layer
I've published a fair amount of static analysis on MCP schemas: distinctive share, input overlap, name distinctiveness, parameter coverage. All of it measures properties of text.
None of it observes a model choosing a tool. Which means every number reports how often a pattern exists, not how often it costs you a call — and the gap between those two is where all the value is.
The probe closes that gap for a pair at a time, cheaply, before you have any production traffic at all. If you maintain an MCP server and you've been meaning to audit your descriptions, this is the thing I'd do first.
I build MCPulse, which reports what models actually do with your tools once real traffic arrives — retries, empty results, first-call success, schema cost. There's also a free schema checker that ranks your tool pairs by input overlap, which is the prefilter step above.
The control-and-probe design came from readers of a previous post. Best methodology suggestion I've had, and it came from people who run servers rather than measure them.
Top comments (0)