We published the scores of 55 public MCP servers, 1,274 tools in all, at forecall.dev/en/servers. Each tool description is scored out of 100 on whether it says what the tool is for, when to use it, its parameters, what it returns, its constraints and an example.
Then we tested whether a low score means models pick the wrong tool. For one-step choices, it didn't: two frontier models picked the right tool 91–99% of the time, low scores or not. This post covers both: what the descriptions leave out, and what that does and doesn't tell you.
Where the 1,274 tools came from
- 46 servers are Smithery's most-used servers, copied from its registry API on October 7, 2026 (UTC). The registry keeps each tool's name, description and input schema, but drops output schemas, annotations and server instructions.
- 9 servers were taken from the running server with
forecall dump: Notion, Playwright, Chrome DevTools, Context7, Filesystem, Memory, Firecrawl, DebugBase and Bhived.
All were scored with the same rules (Forecall's scoring rules v2). Each server's page says which source it came from and when. The pages show scores, issues and tool names, not the descriptions themselves. If you own one of these servers and want it taken down, email support@forecall.dev and it's removed within 7 days.
What the descriptions leave out
885 of 1,274 tools (69%) never say when to use them. The share is the same in both groups: 69% of the registry copies and 71% of the dumped tools.
87 of the 156 dumped tools (56%) don't say what they return, neither in an output schema nor in the description. (I'm leaving the registry copies out of this one, because the registry drops output schemas.)
18 of the 55 servers have at least one pair of near-identical descriptions, 129 pairs in all. Two servers account for most of them: Notion (57 pairs across 24 tools) and Paper Search (29 across 25).
Server averages range from 19.9 to 82, with a median of 44. Across all tools, 497 score under 40 and only 18 score 80 or more.
Does a low score mean models choose wrong? We measured it
Forecall also runs evaluations: it writes requests for each tool and asks models which tool they'd call. We used that to calibrate the static score on 4 servers with 88 tools (Memory, Notion, Playwright and Chrome DevTools), 8 one-step requests per tool, with Claude Sonnet 5.5 and GPT-6.1 Sol:
| Server | Tools | Average score | Right tool (Sonnet / Sol) |
|---|---|---|---|
| Memory | 9 | 43.8 | 97% / 99% |
| Notion | 24 | 33.0 | 96% / 96% |
| Playwright | 25 | 41.9 | 93% / 94% |
| Chrome DevTools | 30 | 45.1 | 99% / 98% |
The correlation between a tool's score and how often models picked it correctly was 0.04 (Spearman). Tools scoring under 40 were picked correctly 95.5% of the time; tools scoring 60–79, 100%.
We then put 25–55 tools from other servers next to each one, to make the choice harder. None of the requests that needed a tool went to the other servers' tools, and accuracy stayed at 91–98%.
So, for frontier models choosing in one step, a short description wasn't the problem: the tool names and schemas carried enough. The misses that did happen were mostly the models doing something sensible first. When a request didn't include a page number, they called list_pages before acting, in 77–87% of those requests in an earlier run. For browser_fill_form, they took a page snapshot first, because the field references come from it.
What the score is for, then
The score says what a description puts in words, not how a model will choose. We changed the site's wording to match: what we used to call "pairs easy to mix up" are now "near-identical descriptions", and every result says what static scoring can't tell.
What's missing from the text may still matter where this test doesn't reach: when the user doesn't have the ID a tool needs, over several steps, for smaller models, and for the people reading your docs. Just don't assume a longer description is better. In an earlier test, rewrites that raised scores (one tool went from 31 to 74) made two models worse (details here). Check rewrites against the models your users run.
Check your own server
- Look it up: forecall.dev/en/servers
- Not listed? If it's a public Streamable HTTP server that needs no sign-in, paste its URL at forecall.dev/en/lint.
- Locally, with nothing sent anywhere:
npx forecall dump -o tools.json -- <command that starts your server>, thennpx forecall lint tools.json. The CLI also reads endpoint lists and search results. - To see how models actually pick your tools, Forecall's evaluations (Pro and Team plans) report the right-tool rate, look-ups first, argument validity and false calls per model.
I'm the author of Forecall. The linter is open source: github.com/forecall/forecall-cli.
Top comments (0)