DEV Community: Hiroki Nomura

I open-sourced a linter for MCP tool descriptions

Hiroki Nomura — Thu, 08 Oct 2026 03:54:27 +0000

AI agents choose an MCP tool by reading three things: its name, its description and its input schema. Many servers write those descriptions like docstrings, for a human who already knows why they opened the file. An agent with thirty tools in its context doesn't know that. It needs to know when to use a tool, what comes back, and which lookalike to use instead.

Forecall's linter checks exactly that, and its code is now public under Apache-2.0: github.com/forecall/forecall-cli.

Try it on a real server

Two commands. The first starts a server and saves its tools/list; the second scores it.

npx forecall dump -o memory.json -- npx -y @modelcontextprotocol/server-memory
npx forecall lint memory.json

Here is part of the report for the official memory server (0.6.3):

Average score     43.8 / 100
Tools             9
Confusable pairs  1

Across the server
  Major     confusable_pair  delete_entities and delete_relations are easy to mix up.
                             Description similarity 0.62 · Name similarity 0.33

  39 / 100  delete_relations
    Purpose 6/20 · When to use 0/20 · Arguments 18/25 · Return value 8/15 · Constraints and side effects 7/10 · Examples 0/10
    Major     restates_name     The description only restates the name (7 words).
    Major     too_short         The description is too short (7 words).
    Major     no_usage_context  Says nothing about when to use it, or when not to.

Two delete tools with similar descriptions, and neither says when to use it. That's the kind of pair where an agent deletes the wrong thing. I've suggested rewrites for them in an issue on the servers repo.

For a remote server, pass its URL instead, with --header "Authorization: Bearer …" if it needs a key:

npx forecall dump https://example.com/mcp -o tools.json

What it checks

Each tool gets a score out of 100, in six parts:

Part	Points	What it looks for
Purpose	20	What the tool does, beyond restating its name
When to use	20	When to use it, and which similar tool to use instead
Arguments	25	Each argument described, typed and marked required
Return value	15	What comes back, in an output schema or the description
Constraints and side effects	10	Preconditions, side effects, auth, rate limits, annotations
Examples	10	An example of a call

Across the server, it flags too many tools, tools that are easy to mix up, and descriptions that share a long paragraph. Every issue has a code and a severity, so you can work through them one by one.

Use it in CI

--fail-under 60 exits with 1 when the average is below 60, and --json prints the report as JSON.

- run: npx forecall dump -o tools.json -- node dist/server.js
- run: npx forecall lint tools.json --fail-under 60

Nothing leaves your machine

lint works offline. The scoring lives in @forecall/lint, a package with no dependencies and no network code. dump is the only command that talks to the network, and only to the server you name; the JSON it writes records where the tools came from, never headers or environment values.

What it can't tell you

The score comes from rules that read the text and the schema. A short description like browser_close can score low even when every model uses it correctly, and a high score doesn't prove that a model will pick the right tool. Use the score to find where to write more. (Forecall's hosted Evaluations run real models against your tools for that. They're paid, and not in this repository.)

What we found with it

We ran it over the 46 most-used servers on Smithery: 769 of 1,118 tools (69%) never say when to use them. The full write-up is here.

Feedback welcome

The rules are at v1, and I'd like to hear where they're wrong. A reader of that write-up already suggested one: when two tools are flagged as lookalikes, check that each description names the other. It's now issue #7. Issues and pull requests are welcome.

I'm the author of Forecall.

We checked 46 popular MCP servers on Smithery: 7 in 10 tools never say when to use them

Hiroki Nomura — Thu, 08 Oct 2026 02:55:15 +0000

We took 46 of the most-used MCP servers on Smithery and checked the descriptions of all 1,118 of their tools. 769 tools (69%) never say when to use them. 507 (45%) never say in their description what they return. And 15 servers have tools whose descriptions are so alike that an agent can mix them up: 90 such pairs in all.

The 69% is exactly what we found last time, when we checked 8 public servers with 126 tools and 87 of them lacked it. Nine times the sample, the same share.

The checks were done with Forecall, a static linter for MCP tool descriptions that I build.

How we checked: the top 50 servers on Smithery, judged by their text only

On October 8, 2026, we took the first 500 servers in Smithery's public registry (in the order its API returns), removed duplicates, picked the 50 with the most uses, and collected their tool definitions. We scored them with Forecall's CLI (forecall 0.3.2). Four of the 50 have more than 200 tools, the CLI's limit, so we left them out: 46 servers and 1,118 tools remain.

One caveat. The tool definitions Smithery's API returns leave out the input schemas' required lists and any output schemas, so Forecall's overall scores come out lower than they should. This article does not use the overall scores. It counts only the findings that depend on the description text alone: whether a tool says when to use it, whether its description says what it returns, whether the description is too short, and pairs of tools whose descriptions are alike.

7 in 10 tools never say when to use them

Finding	Tools	Share
No "when to use"	769	69%
Description doesn't say what it returns	507	45%
Too short, or restates the name	101	9%

By server it is starker. In 16 of the 46 servers, not one tool says when to use it. Count the servers where 80% or more lack it, and you have half of them: 23.

"When to use" is what decides between lookalike tools. Without it, a model picks by the tools' names and the feel of their descriptions.

The same notice on every tool makes the tools indistinguishable

PubMed has the most lookalike pairs. It has only 7 tools, yet all 21 pairs are flagged as easy to mix up. Among the 12 most similar pairs, the descriptions share 73% to 94% of their words.

The cause is a long notice pasted into all 7 descriptions:

IMPORTANT - PubMed Database Scope: This server provides access to PubMed, which ONLY indexes biomedical and life sciences literature including: …

What sets each tool apart, its first sentence ("search articles", "get metadata", "find related articles" and so on), is buried in the shared paragraph.

A note that applies to the whole server belongs once in the server's instructions, which the client receives at initialization, not in every tool's description. A tool's description should say what only that tool does.

"Send a reply" and "draft a reply" are written almost the same way

Gmail has 7 lookalike pairs. The closest two share 92% of their words:

Tool	Start of its description
Gmail_ReplyToEmail	Send a reply to an email message, optionally with one or more file attachments.
Gmail_WriteDraftReplyEmail	Compose a draft reply to an email message, optionally with one or more file attachments.

After that, both go on with the same paragraph about attaching files. The difference is a few words: "Send" against "Compose a draft".

Mix these two up, and an agent meant to write a draft sends the email. Tools that do something you can't undo, above all, should say in one sentence how to choose: "To prepare a reply without sending it, use Gmail_WriteDraftReplyEmail."

A good example: Brave Search says how its tools connect

Brave Search's descriptions, by contrast, spell out how the tools relate. Here is brave_web_search:

Performs web searches using the Brave Search API and returns comprehensive search results with rich metadata. To chain into local-POI enrichment, pass result_filter=locations and feed the resulting locations.results[].id values into brave_local_search.

It says what comes back and which value to pass to which tool next, so an agent can use several tools in sequence. All 8 of Brave Search's tools say in their descriptions what they return, and it has no lookalike pairs.

How to fix it: one sentence each for three things

From these results, this is the order to fix descriptions in:

Say when to use the tool. "Use this when …", and if there is a lookalike, "for …, use X instead."
Move shared notes to the server's instructions. The same paragraph on every tool buries what tells them apart.
Put the difference between lookalikes into words. Send or draft? Delete or remove? Say it plainly, especially for actions you can't undo.

To check your own server, run these two commands. The first gets the tools/list from your server, the second scores it. Both run on your machine and send the results nowhere.

npx forecall dump -o tools.json -- <command that starts your server>
npx forecall lint tools.json

For a Streamable HTTP server, pass its URL instead: npx forecall dump https://example.com/mcp > tools.json. The CLI is open source: https://github.com/forecall/forecall-cli

The weakness doesn't shrink with scale

With 8 servers or 46, 7 in 10 tools never said when to use them, popular servers included. The fix is small: one sentence on when to use the tool and one on how it differs from its lookalikes leave an agent far less room to pick the wrong one.

We'll keep this study going with new data. Next, we'll score the remote servers in the official MCP Registry, schemas included.

The same study is also on Qiita, in Japanese: https://qiita.com/nomunomu0504/items/b2eab1960a7aad38b59d