We took 46 of the most-used MCP servers on Smithery and checked the descriptions of all 1,118 of their tools. 769 tools (69%) never say when to use...
For further actions, you may consider blocking this person and/or reporting abuse
ha, a send tool and a draft tool sharing 92% of their words is the scariest number in here. In our experience at ApyHub we went the other way: our MCP has 4 tools (catalog, search, spec, call), so there's no sibling to mix up, and endpoint text only shows up after the agent searches (I co-founded it). Would Forecall score a setup like that, or does it only look at the tool list?
Good question. The linter only reads tools/list, so it would score your four tools (catalog, search, spec, call) and check that they're easy to tell apart. It doesn't see the endpoint text that search and spec return at run time, so it can't judge how well that guides the agent's next call. With four tools the lookalike risk is small; what matters more in that design is whether each description says what comes back and which tool to call next with it, as Brave Search does in the article. Evaluations also start from tools/list: they'd measure which of the four a model picks and whether its arguments fit, not which endpoint it picks after a search. Scoring dynamic discovery like yours would need a different kind of test.
yeah, "say which tool to call next" is a good rule, I'll run our 4 descriptions against it. In our experience at ApyHub the hard part sits one step later, when the agent picks an endpoint from what search returns. Would scoring search results ever fit in Forecall, or is that a different tool?
Not today. Forecall reads tools/list, and Evaluations measure which tool a model calls from that list, so the step where the agent picks an endpoint out of search results is outside both.
One thing that works now: if search returns endpoint entries with a name, a summary and parameters, you can put a sample of them into tools/list shape ({ name, description, inputSchema }) and run
npx forecall linton that file. It scores each entry's text the same way and flags entries whose descriptions are nearly identical. That tells you what the entries leave out, not which endpoint a model would pick; measuring the pick would need an eval that includes the search step, which we don't run. I've noted it as a request.Update: your question turned into a feature.
npx forecall lint(0.8.0) and the web linter now read a list of candidates that isn't tools/list:{"endpoints": [...]}(orresults,items,entries, or a bare array), where each entry has a name, operationId, id, title or method + path, plus a description, summary, text or snippet. Parameters come from a schema or an OpenAPI-styleparametersarray, so there's no need to convert anything by hand:npx forecall lint endpoints.jsonSame caveat as before: the score says what each entry's text leaves out, not which endpoint a model picks. For the pick, you can register the list as a server and run an evaluation, which measures which entry a model calls for each request (Pro and Team plans). What it still doesn't do is call your search tool and choose the next step from what it actually returned; that's logged as a request.
Docs: forecall.dev/en/docs/tools-list#en...
A timestamp will not separate a revised tool contract from an unchanged one republished, which matters for the registry pass you have announced. In a public append-only event log I parse, the self-declared capability directory holds 1,633 declaration rows carrying 1,644 entries, and those collapse to 27 distinct names. 22 of the 27 have exactly one definition hash across every row they ever appeared in.
verify.result.basicsits in 136 rows from a single key, byte-identical every time. Version labels do not rescue it in either direction: 24 of the 27 declare no version at all, andtransform.text.democarries 10 distinct definitions under two labels ("1.0", and absent). Hash each normalized tool definition and diff on that hash between observations, and a server can no longer earn "updated recently" by reposting the same manifest.The 1,118 is a count of declared tools, and declared surface can sit very far from exercised surface. In this one deployment only three distinct capability names ever appear as a requested capability, and 1,520 of the 1,524 parsed requests ask for the same one. Of 1,622 delivery rows, 1,608 resolve back to a request, and they reach those same 3 names. 20 of the 27 declared names belong to keys that never delivered anything. One small deployment proves nothing about Smithery's population, and server-level use counts are not tool-level call counts, so the measurement I would want next is a usage weight on your 769, telling you whether under-described tools are the ones models actually reach for or whether they sit in a tail nothing calls.
One record-shape detail bears on your Gmail pair. Delivery rows in this log have no capability-name field, so the name attached to a completed call is read off the request row, which is the caller's own string. A send-instead-of-draft confusion would be reconstructable only from the side that chose. Does the official MCP Registry expose anything content-addressed per tool definition, or only a version string plus a publish timestamp?
Thanks, a lot here. On versions: Forecall's own registry already works the way you describe. Each tools/list gets a SHA-256 of a canonical JSON, and each tool its own; a server gets a new version only when that hash changes, so reposting the same manifest doesn't count as an update. That's the right basis for the registry pass too.
On your question: as far as I can see, the official MCP Registry has nothing per tool. A version's server.json has the name, version, description and remotes or packages, but no tools, and the registry adds status, publishedAt, updatedAt and isLatest. To get tool definitions you have to connect and call tools/list.
On declared vs exercised surface: agreed, 769 counts declared tools. We don't have per-tool call counts for these servers, so a usage weight isn't something we can add yet.
Moving the PubMed scope notice into the server instructions does more than remove a repeated paragraph. In Claude Code, where tool search is the default, the docs say only tool names and server instructions load at session start, and describe the instructions as what helps the model decide when to search for a server's tools. So "when to use" is asked twice, from two texts: whether to reach into this server at all, decided on names and instructions, and which tool to call, decided on descriptions read only after that first choice.
The PubMed notice answers the first question in the place where the second one is asked. "ONLY indexes biomedical and life sciences literature" says when not to use the server and nothing about "search articles" against "get metadata", which is why all it gives the lookalike check is shared words. It also means the 769 can hold two different absences: a tool with no "when to use" on a server whose instructions name its scope is missing one answer, and the same tool on a server with empty instructions is missing both, with only tool names in context under tool search.
Since the linter reads only tools/list, a server that follows the second fix looks the same as one that deleted the note. If the registry pass connects to each remote server for its schemas, the same connection carries whatever instructions the server sets, and recording them next to tools/list would split a count like the 769 by whether the server answers the first question anywhere. Claude Code also cuts each server's instructions at 2,048 characters by default, the same cap as a tool description, so the scope sentence has to come first there too.
Good point, and the two-step framing is clearer than what I wrote. The PubMed note answers "should I reach into this server at all", which belongs in the instructions; the lookalike check is about "which of these tools". You're right that the linter can't tell a server that moved the note into its instructions from one that deleted it: it reads tools/list only.
forecall dumpalready keeps the server's instructions next to tools/list, so in the registry pass we can record them and split the 769 by whether the server says when to use it anywhere. Thanks for the note on the 2,048-character cap; putting the scope sentence first is a good rule in both places.The root issue is that server authors still write MCP descriptions as API docstrings for humans rather than routing discriminators for a planner.
When a human reads a docstring, they already know why they opened that file. When an agent loads thirty tools into context, the failure mode is almost never misunderstanding what the function does. The failure mode is symmetric ambiguity with an adjacent tool. If search_code and grep_files both say they find text across the repository, the model effectively rolls dice on ambiguous prompts.
Flagging the 90 overlapping pairs is the most practical part of this audit. The logical next check for Forecall would be scanning for explicit negative boundaries, where descriptions name what they do not handle and pointer tokens to the sibling tool meant for that case. In my agent runs, adding a single negative pointer usually cuts down more tool-choice thrashing than expanding the parameter schema.
Thanks, this matches what we saw. On negative boundaries: Forecall already gives partial credit for them under "when to use". I tried your example: two tools that only said "Finds text across the repository" scored 0/20 on usage and were flagged as a lookalike pair. Adding one sentence to each ("to match an exact string or regex, use grep_files instead" / "not for semantic search, which search_code handles") took both to 15/20, and the pair was no longer flagged.
What it doesn't check yet is whether lookalike tools point at each other: when two tools are flagged as a pair, does each description name the other? That's a good rule. I'm taking it to our list.
Update: it's now an issue, with you credited: github.com/forecall/forecall-cli/i...
Follow-up: it's live. When two tools are flagged as a lookalike pair, the finding now tells you to say in each description when to use the other one instead, in the CLI (forecall 0.4.2) and on forecall.dev. It's advice rather than a new check, because the pointer sentence is usually what makes the pair stop being flagged. Thanks again for the idea: github.com/forecall/forecall-cli/p...