An agent calling your MCP server never sees your code. It sees tool names, descriptions and parameter docs, and decides from those alone which tool to call and what to send. When a description is missing or wrong, the agent picks the wrong tool or sends bad arguments, and nothing in your own tests notices.
I built mcp-doctor to check for that. It's a static linter for MCP server source (Python, TypeScript, Go) that grades quality and security separately. It never starts the server. This week it also ships as a Claude Code plugin, and this post is about the part I spent the most time on: testing that the plugin doesn't make things up.
Why a plugin, not just the CLI
The CLI is good at the mechanical half: finding tools, spotting a parameter with no description, an optional argument typed non-null, a print() on a stdio server. It can't write a good tool description, because that needs someone to read what the tool actually does.
An agent can do that half. So the plugin is a skill that:
- runs the CLI and reports both grades plus at most five findings, each phrased as what goes wrong for the agent ("the model can't tell when to call
run"), not as a rule name; - shows the mechanical fixes as a diff and applies them when you say yes;
- reads each tool's code and writes the descriptions, parameter docs and error messages the CLI can't;
- re-scans and shows the grades before and after.
Install:
/plugin install mcp-doctor --marketplace vishalhabib99/mcp-doctor
Then ask "check my MCP server".
The risk: a confident, wrong description
Handing description-writing to a model is risky in a specific way. A model asked to "add a description" to a tool called run that takes a cmd argument will happily write "Executes a shell command". That reads well and could be completely false.
The sample server in the repo has exactly that tool. run(cmd) ignores cmd and returns the string "ran". A false description there is worse than none: it tells the next agent this tool can run commands, so the agent will try.
So the skill's instructions say: read the tool's code before writing anything, and never describe behavior the code doesn't have. Instructions are cheap, though. I wanted a test that fails if the model ignores them.
The eval suite
The plugin ships with five cases, run with claude plugin eval:
| Case | What it checks |
|---|---|
scan-bad-server |
The skill fires, runs the CLI, reports Quality 13% F and Security 100% A separately, edits nothing |
fix-bad-server |
Applies fixes, shows before and after, and does not describe run as executing a command |
clean-good-server |
Reports A / A on a clean server and proposes no fixes |
not-mcp-code |
Doesn't fire on unrelated Python work |
mcp-config-not-server |
Doesn't fire when you want to use an MCP server, not build one |
The last two matter as much as the first three. A skill that fires on every Python file is noise, and one that "fixes" a clean server can't be trusted.
There's also a rule for when the CLI can't run (not installed, no network, Bash not allowed): say so in the first line and never state a grade the CLI didn't produce. The eval sandbox blocks the installed CLI, so that path got tested whether I liked it or not. The graders accept either the real grades or a plain "the scanner didn't run" with no invented number.
Results
Last full run: 3 runs per case, with and without the plugin.
- The skill fired in 9 of 9 MCP runs and 0 of 6 unrelated ones.
- 13 of 15 runs passed every grader. Overall score was 0.94 with the plugin, 0.63 without it.
- Both misses were judge votes on runs where the sandbox blocked the CLI and the skill said so without inventing a grade. I count that as the honesty rule working and the judge being strict, but they're still misses.
In a normal session with the CLI available, the skill took the sample server from 13% to 42%. The rest of the gap is a missing README, LICENSE, tests and pinned dependencies, which it listed but didn't create, because a LICENSE is your call, not the agent's. It described run as a placeholder that ignores cmd. On the clean server it reported A / A and changed nothing.
What's underneath
The plugin is only as good as the scanner. The CLI finds 15,755 of 15,977 tools across 805 public servers, with every miss listed in coverage.md. Running it on real servers has turned up 104 bugs in mcp-doctor itself, all fixed, and 22 fixes I've sent to other people's MCP servers have been merged.
Takeaway
If you ship an agent skill that writes anything a later agent will rely on, test the failure where it writes something plausible and false. Put a tool in your fixtures whose name invites a wrong description, and fail the run when the model takes the bait.
Bug reports are welcome, especially servers where it gets the tool count wrong: github.com/vishalhabib99/mcp-doctor
Top comments (0)