We checked 46 popular MCP servers on Smithery: 7 in 10 tools never say when to use them

We took 46 of the most-used MCP servers on Smithery and checked the descriptions of all 1,118 of their tools. 769 tools (69%) never say when to use them. 507 (45%) never say in their description what they return. And 15 servers have tools whose descriptions are so alike that an agent can mix them up: 90 such pairs in all.

The 69% is exactly what we found last time, when we checked 8 public servers with 126 tools and 87 of them lacked it. Nine times the sample, the same share.

The checks were done with Forecall, a static linter for MCP tool descriptions that I build.

How we checked: the top 50 servers on Smithery, judged by their text only

On October 8, 2026, we took the first 500 servers in Smithery's public registry (in the order its API returns), removed duplicates, picked the 50 with the most uses, and collected their tool definitions. We scored them with Forecall's CLI (forecall 0.3.2). Four of the 50 have more than 200 tools, the CLI's limit, so we left them out: 46 servers and 1,118 tools remain.

One caveat. The tool definitions Smithery's API returns leave out the input schemas' required lists and any output schemas, so Forecall's overall scores come out lower than they should. This article does not use the overall scores. It counts only the findings that depend on the description text alone: whether a tool says when to use it, whether its description says what it returns, whether the description is too short, and pairs of tools whose descriptions are alike.

7 in 10 tools never say when to use them

Finding	Tools	Share
No "when to use"	769	69%
Description doesn't say what it returns	507	45%
Too short, or restates the name	101	9%

By server it is starker. In 16 of the 46 servers, not one tool says when to use it. Count the servers where 80% or more lack it, and you have half of them: 23.

"When to use" is what decides between lookalike tools. Without it, a model picks by the tools' names and the feel of their descriptions.

The same notice on every tool makes the tools indistinguishable

PubMed has the most lookalike pairs. It has only 7 tools, yet all 21 pairs are flagged as easy to mix up. Among the 12 most similar pairs, the descriptions share 73% to 94% of their words.

The cause is a long notice pasted into all 7 descriptions:

IMPORTANT - PubMed Database Scope: This server provides access to PubMed, which ONLY indexes biomedical and life sciences literature including: …

What sets each tool apart, its first sentence ("search articles", "get metadata", "find related articles" and so on), is buried in the shared paragraph.

A note that applies to the whole server belongs once in the server's instructions, which the client receives at initialization, not in every tool's description. A tool's description should say what only that tool does.

"Send a reply" and "draft a reply" are written almost the same way

Gmail has 7 lookalike pairs. The closest two share 92% of their words:

Tool	Start of its description
Gmail_ReplyToEmail	Send a reply to an email message, optionally with one or more file attachments.
Gmail_WriteDraftReplyEmail	Compose a draft reply to an email message, optionally with one or more file attachments.

After that, both go on with the same paragraph about attaching files. The difference is a few words: "Send" against "Compose a draft".

Mix these two up, and an agent meant to write a draft sends the email. Tools that do something you can't undo, above all, should say in one sentence how to choose: "To prepare a reply without sending it, use Gmail_WriteDraftReplyEmail."

A good example: Brave Search says how its tools connect

Brave Search's descriptions, by contrast, spell out how the tools relate. Here is brave_web_search:

Performs web searches using the Brave Search API and returns comprehensive search results with rich metadata. To chain into local-POI enrichment, pass result_filter=locations and feed the resulting locations.results[].id values into brave_local_search.

It says what comes back and which value to pass to which tool next, so an agent can use several tools in sequence. All 8 of Brave Search's tools say in their descriptions what they return, and it has no lookalike pairs.

How to fix it: one sentence each for three things

From these results, this is the order to fix descriptions in:

1. Say when to use the tool. "Use this when …", and if there is a lookalike, "for …, use X instead."
2. Move shared notes to the server's instructions. The same paragraph on every tool buries what tells them apart.
3. Put the difference between lookalikes into words. Send or draft? Delete or remove? Say it plainly, especially for actions you can't undo.

To check your own server, run these two commands. The first gets the tools/list from your server, the second scores it. Both run on your machine and send the results nowhere.

npx forecall dump -o tools.json -- <command that starts your server>
npx forecall lint tools.json

For a Streamable HTTP server, pass its URL instead: npx forecall dump https://example.com/mcp > tools.json. The CLI is open source: https://github.com/forecall/forecall-cli

The weakness doesn't shrink with scale

With 8 servers or 46, 7 in 10 tools never said when to use them, popular servers included. The fix is small: one sentence on when to use the tool and one on how it differs from its lookalikes leave an agent far less room to pick the wrong one.

We'll keep this study going with new data. Next, we'll score the remote servers in the official MCP Registry, schemas included.

The same study is also on Qiita, in Japanese: https://qiita.com/nomunomu0504/items/b2eab1960a7aad38b59d

Comments

[Nikolas Dimitroulakis]

ha, a send tool and a draft tool sharing 92% of their words is the scariest number in here. In our experience at ApyHub we went the other way: our MCP has 4 tools (catalog, search, spec, call), so there's no sibling to mix up, and endpoint text only shows up after the agent searches (I co-founded it). Would Forecall score a setup like that, or does it only look at the tool list?

[Hiroki Nomura]

Good question. The linter only reads tools/list, so it would score your four tools (catalog, search, spec, call) and check that they're easy to tell apart. It doesn't see the endpoint text that search and spec return at run time, so it can't judge how well that guides the agent's next call. With four tools the lookalike risk is small; what matters more in that design is whether each description says what comes back and which tool to call next with it, as Brave Search does in the article. Evaluations also start from tools/list: they'd measure which of the four a model picks and whether its arguments fit, not which endpoint it picks after a search. Scoring dynamic discovery like yours would need a different kind of test.

[Nikolas Dimitroulakis]

yeah, "say which tool to call next" is a good rule, I'll run our 4 descriptions against it. In our experience at ApyHub the hard part sits one step later, when the agent picks an endpoint from what search returns. Would scoring search results ever fit in Forecall, or is that a different tool?

[Hiroki Nomura]

Not today. Forecall reads tools/list, and Evaluations measure which tool a model calls from that list, so the step where the agent picks an endpoint out of search results is outside both.

One thing that works now: if search returns endpoint entries with a name, a summary and parameters, you can put a sample of them into tools/list shape ({ name, description, inputSchema }) and run npx forecall lint on that file. It scores each entry's text the same way and flags entries whose descriptions are nearly identical. That tells you what the entries leave out, not which endpoint a model would pick; measuring the pick would need an eval that includes the search step, which we don't run. I've noted it as a request.

[Hiroki Nomura]

Update: your question turned into a feature. npx forecall lint (0.8.0) and the web linter now read a list of candidates that isn't tools/list: {"endpoints": [...]} (or results, items, entries, or a bare array), where each entry has a name, operationId, id, title or method + path, plus a description, summary, text or snippet. Parameters come from a schema or an OpenAPI-style parameters array, so there's no need to convert anything by hand:

{"endpoints": [{"method": "get", "path": "/users/{id}", "summary": "Read one user."}, {"method": "delete", "path": "/users/{id}", "summary": "Remove a user."}]}

npx forecall lint endpoints.json

Same caveat as before: the score says what each entry's text leaves out, not which endpoint a model picks. For the pick, you can register the list as a server and run an evaluation, which measures which entry a model calls for each request (Pro and Team plans). What it still doesn't do is call your search tool and choose the next step from what it actually returned; that's logged as a request.

Docs: forecall.dev/en/docs/tools-list#en...

[ANP2 Network]

A timestamp will not separate a revised tool contract from an unchanged one republished, which matters for the registry pass you have announced. In a public append-only event log I parse, the self-declared capability directory holds 1,633 declaration rows carrying 1,644 entries, and those collapse to 27 distinct names. 22 of the 27 have exactly one definition hash across every row they ever appeared in. verify.result.basic sits in 136 rows from a single key, byte-identical every time. Version labels do not rescue it in either direction: 24 of the 27 declare no version at all, and transform.text.demo carries 10 distinct definitions under two labels ("1.0", and absent). Hash each normalized tool definition and diff on that hash between observations, and a server can no longer earn "updated recently" by reposting the same manifest.

The 1,118 is a count of declared tools, and declared surface can sit very far from exercised surface. In this one deployment only three distinct capability names ever appear as a requested capability, and 1,520 of the 1,524 parsed requests ask for the same one. Of 1,622 delivery rows, 1,608 resolve back to a request, and they reach those same 3 names. 20 of the 27 declared names belong to keys that never delivered anything. One small deployment proves nothing about Smithery's population, and server-level use counts are not tool-level call counts, so the measurement I would want next is a usage weight on your 769, telling you whether under-described tools are the ones models actually reach for or whether they sit in a tail nothing calls.

One record-shape detail bears on your Gmail pair. Delivery rows in this log have no capability-name field, so the name attached to a completed call is read off the request row, which is the caller's own string. A send-instead-of-draft confusion would be reconstructable only from the side that chose. Does the official MCP Registry expose anything content-addressed per tool definition, or only a version string plus a publish timestamp?

[Hiroki Nomura]

Thanks, a lot here. On versions: Forecall's own registry already works the way you describe. Each tools/list gets a SHA-256 of a canonical JSON, and each tool its own; a server gets a new version only when that hash changes, so reposting the same manifest doesn't count as an update. That's the right basis for the registry pass too.

On your question: as far as I can see, the official MCP Registry has nothing per tool. A version's server.json has the name, version, description and remotes or packages, but no tools, and the registry adds status, publishedAt, updatedAt and isLatest. To get tool definitions you have to connect and call tools/list.

On declared vs exercised surface: agreed, 769 counts declared tools. We don't have per-tool call counts for these servers, so a usage weight isn't something we can add yet.

[Edward Izgorodin]

Moving the PubMed scope notice into the server instructions does more than remove a repeated paragraph. In Claude Code, where tool search is the default, the docs say only tool names and server instructions load at session start, and describe the instructions as what helps the model decide when to search for a server's tools. So "when to use" is asked twice, from two texts: whether to reach into this server at all, decided on names and instructions, and which tool to call, decided on descriptions read only after that first choice.

The PubMed notice answers the first question in the place where the second one is asked. "ONLY indexes biomedical and life sciences literature" says when not to use the server and nothing about "search articles" against "get metadata", which is why all it gives the lookalike check is shared words. It also means the 769 can hold two different absences: a tool with no "when to use" on a server whose instructions name its scope is missing one answer, and the same tool on a server with empty instructions is missing both, with only tool names in context under tool search.

Since the linter reads only tools/list, a server that follows the second fix looks the same as one that deleted the note. If the registry pass connects to each remote server for its schemas, the same connection carries whatever instructions the server sets, and recording them next to tools/list would split a count like the 769 by whether the server answers the first question anywhere. Claude Code also cuts each server's instructions at 2,048 characters by default, the same cap as a tool description, so the scope sentence has to come first there too.

[Hiroki Nomura]

Good point, and the two-step framing is clearer than what I wrote. The PubMed note answers "should I reach into this server at all", which belongs in the instructions; the lookalike check is about "which of these tools". You're right that the linter can't tell a server that moved the note into its instructions from one that deleted it: it reads tools/list only. forecall dump already keeps the server's instructions next to tools/list, so in the registry pass we can record them and split the 769 by whether the server says when to use it anywhere. Thanks for the note on the 2,048-character cap; putting the scope sentence first is a good rule in both places.

[Reid Marlow]

The root issue is that server authors still write MCP descriptions as API docstrings for humans rather than routing discriminators for a planner.

When a human reads a docstring, they already know why they opened that file. When an agent loads thirty tools into context, the failure mode is almost never misunderstanding what the function does. The failure mode is symmetric ambiguity with an adjacent tool. If search_code and grep_files both say they find text across the repository, the model effectively rolls dice on ambiguous prompts.

Flagging the 90 overlapping pairs is the most practical part of this audit. The logical next check for Forecall would be scanning for explicit negative boundaries, where descriptions name what they do not handle and pointer tokens to the sibling tool meant for that case. In my agent runs, adding a single negative pointer usually cuts down more tool-choice thrashing than expanding the parameter schema.

[Hiroki Nomura]

Thanks, this matches what we saw. On negative boundaries: Forecall already gives partial credit for them under "when to use". I tried your example: two tools that only said "Finds text across the repository" scored 0/20 on usage and were flagged as a lookalike pair. Adding one sentence to each ("to match an exact string or regex, use grep_files instead" / "not for semantic search, which search_code handles") took both to 15/20, and the pair was no longer flagged.

What it doesn't check yet is whether lookalike tools point at each other: when two tools are flagged as a pair, does each description name the other? That's a good rule. I'm taking it to our list.

[Hiroki Nomura]

Update: it's now an issue, with you credited: github.com/forecall/forecall-cli/i...

[Hiroki Nomura]

Follow-up: it's live. When two tools are flagged as a lookalike pair, the finding now tells you to say in each description when to use the other one instead, in the CLI (forecall 0.4.2) and on forecall.dev. It's advice rather than a new check, because the pointer sentence is usually what makes the pair stop being flagged. Thanks again for the idea: github.com/forecall/forecall-cli/p...