Your MCP server's tool schema is the entire interface a model has. No README, no repo, no idea what you meant — just the JSON from tools/list.
We read that JSON for 4,951 public servers: 87,146 tools and 270,487 parameters. Here's what's in it.
The short version: one tool in six carries a description containing no word that distinguishes it from a sibling tool on the same server. On servers with more than sixty tools, it's nearly one in three.
What this is, and what it isn't
This post is about what models are given. It is not about what they do with it.
We did not run a model against these servers. We never observed a tool selection, an argument, or a retry, so nothing here can tell you how often models actually get it wrong. Every number below is a property of a schema sitting still.
We're drawing that line hard because it's the line the interesting claim sits on. "A model has nothing to discriminate on" is a fact about a schema. "A model therefore picks wrong 30% of the time" is a fact about traffic, and we don't have it.
How we did it
We took the tool schemas from the Smithery registry, which stores the tools/list response for every server it has scanned — inputSchema included. That's byte-for-byte the JSON a model receives.
The obvious alternative was to boot each server in a container and call tools/list ourselves. We didn't, and the reason is the finding underneath the method: a large share of public MCP servers won't start without real credentials. The set that boots cleanly on a machine with no API keys isn't a random subset of anything.
Of 5,123 servers in the frame, 154 detail requests failed, 5 had never been scanned, and 13 published no tools at all. That leaves 4,951 servers with schemas.
The check that nearly ended the study
Before any of this counts for anything, one question has to be answered: does the registry hand back the schema the server actually serves?
We had reason to think it might not. Across the first 1,377 tools we collected, not one carried a top-level required array. Real MCP servers mark parameters required constantly.
So we booted the four official reference servers locally — they need no credentials and no network — took their real tools/list over stdio, and diffed field by field.
Everything survives except required, which is stripped. On one server that's 27 of 37 tools with a required array on the real server, and 0 via the API.
So this study measures nothing about required-versus-optional parameters. If you're doing your own analysis on registry data, that field is not there, and it does not announce itself.
1. One parameter in five has no description at all
59,038 of 270,487 parameters — 21.8% — ship with no description. They appear on 33.7% of servers.
A parameter with no description is a parameter the model guesses at. It has the name, it has the type, and that's the entire brief. Sometimes the name carries it: query on a search tool isn't mysterious. Often it doesn't — we found plenty of bare id, type, mode and filter parameters with nothing to say which of several plausible things they meant.
The striking part is the contrast with tool descriptions. Only 0.4% of tools have no description. Authors describe the tool and forget the arguments — which is understandable, because the tool is the thing you're thinking about when you write it, and the arguments are the thing the model has to fill in.
2. One tool in six has nothing to tell it apart from its neighbour
For every tool, we took the content words in its description and asked how many appear in no other tool's description on the same server. Call it the tool's distinctive share.
- The median tool's description is 26% distinctive. Three-quarters of the words it spends are words its siblings also use.
- 26.7% of tools are under 10% distinctive.
- 17.4% are exactly zero. 23.1% of servers have at least one.
Zero doesn't mean the description is bad. Here are four from one widely-installed Gmail server:
Gmail_DeleteDraftEmail "Delete a draft email using the Gmail API."
Gmail_SendDraftEmail "Send a draft email using the Gmail API."
Gmail_ListLabels "List all the labels in the user's mailbox."
Gmail_SearchThreads "Search for threads in the user's mailbox."
Every one of those is clear, correct English. Every one is also built entirely from words the other tools use — delete, draft, email, gmail, api, list, search, threads, mailbox all recur across the set. The description tells you what the tool does. It doesn't tell you what this tool does and the others don't, and that second thing is the one a model needs when it's choosing.
The pattern that produces the most extreme cases is shared boilerplate. One server appends the same 51-word context block to all 275 of its tools. "Create a new NFT collection" and "Purchase an NFT from a listing" differ by 8 words out of 59. That block was added deliberately, to help.
3. It gets worse the more tools you ship
| Tools on the server | Servers | No description | Zero-distinctive tools |
|---|---|---|---|
| 1–3 | 1,256 | 14.8% | 0.5% |
| 4–7 | 1,292 | 22.4% | 1.6% |
| 8–15 | 1,044 | 21.9% | 4.3% |
| 16–30 | 767 | 24.8% | 7.7% |
| 31–60 | 377 | 22.2% | 16.3% |
| 61+ | 215 | 20.5% | 31.3% |
Description collision rises monotonically and by a factor of sixty. Some of that is arithmetic — more tools means more chances for two to collide — but it's also the point at which authors start generating descriptions from a template, and a template is a machine for producing tools that read alike.
And notice the column that doesn't move. Missing parameter descriptions sit between 20% and 25% at every size above the smallest bucket. It's not a scale problem; it's a habit. The two failures are independent, which means shipping fewer tools won't fix your undescribed parameters and writing better descriptions won't fix your collisions.
4. The median tool list costs about 1,250 tokens before anyone asks a question
Tool schemas are sent on every connection, whether or not a single tool gets called.
- Median server: 4,991 bytes, roughly 1,250 tokens
- 90th percentile: 32,636 bytes, roughly 8,200 tokens
- Largest in the corpus: 1,145,575 bytes — on the order of 280,000 tokens of schema, from one server, before the conversation starts
Median tools per server is 7 and the mean is 17.6. One server publishes 2,530 tools.
5. Some parameters name their valid values and then don't enforce them
8.0% of all parameters carry an enum. What you can find from outside is the case where the author wrote the values down in prose and left the schema as an open string:
outcome_attribution "Attribution type for the outcomes.
Valid values: "direct", "influenced",
"unattributed", "total"."
commitment "Optional processed|confirmed|finalized commitment"
895 parameters, on 4.6% of servers. That's a floor rather than an estimate: it only catches authors who documented the set.
Worth reporting how we got there. Our first version of this measurement said 108 hits in a 40-server sample. It was matching text like Filter by line (e.g. "1", "A", "F"), which is an illustration, not a closed set. Requiring explicit closed-set language took that sample from 108 to 9, and all nine were real.
One signal we expected to find and didn't: duplicate tool names within a server, on 0.2% of servers. Effectively nobody does this. If it's on your review checklist, take it off.
What we couldn't see
This is static analysis, and we never observed a single real request to any of these servers.
That means we missed everything that only shows up under traffic. The tool that works in isolation but gets called in the wrong order. The parameter that's fine until someone phrases a request unusually. The retry loop that only triggers on a specific error path.
We also can't tell you the thing you most want to know, which is how much of this matters. A zero-distinctive description might cost nothing when the tool name is unambiguous, and a great deal when it isn't.
One more hole, found by a server author after publication: the 21.8% measures absence only. A parameter described as "query: The query" counts as described and passes. Restating the parameter name is the more common failure in his experience, and it passes every linter. So that number is a floor too.
If you maintain a server
Three things, ordered by how common the problem is in the data. None of them changes your server's behaviour — they're all changes to the text a model reads.
Describe every parameter. The most common gap by a distance, and it doesn't get better at any size. If you do one thing, do this one.
Make each description say what the others don't. Not "is it clear" — is it clear which of my tools this is. Read your tool list as one block and ask what a reader with only that block would use to choose.
If you have more than about thirty tools, audit for collisions specifically. Above that size, roughly one tool in six has no distinguishing word, rising to one in three past sixty.
The data
Analysis scripts and raw data: https://github.com/getmcpulse/mcp-schema-study
Every figure is in analysis.json, with the servers and tools behind each one in examples.json. The collector rebuilds the whole corpus from a public API in about twenty minutes.
There's also a browser-based checker that runs these measurements on your own tools/list — paste the JSON, get your numbers scored against the corpus: https://getmcpulse.com/check
We ran this because we're building MCPulse, an SDK that reports what actually happens under real traffic. Everything in this post came from outside the server, which is exactly its limit. A schema can tell you a model has nothing to choose on. Only traffic can tell you whether it chose wrong.
Originally published at getmcpulse.com.
Top comments (1)
The strongest part is the line between schema evidence and behavior claims. The stripped required arrays also show why registry data needs provenance of its own: a consumer can receive valid JSON that is no longer the interface the server authored. I would want future measurements to fingerprint the live tools/list response and the registry representation separately.