Originally published at tengli.dev
Your MCP server can be 100% spec-compliant and still be unusable by an agent.
The Model Context Protocol spec tells you how to transport tools: JSON-RPC framing, capability negotiation, schema shapes. It says nothing about whether a model can actually use what you serve — whether it picks the right tool out of your catalog, fills the arguments correctly, or burns 8k tokens parsing your schemas on every single request.
I integrate first- and third-party MCP connectors into a production AI agent for a living, and I kept seeing the same failure: servers that pass every compliance check, yet the model calls the wrong tool, hallucinates arguments, or ignores the tool entirely. The problems were never in the protocol layer. They were in the parts no one lints: descriptions, naming, schema design.
So I wrote mcpgrade — a Lighthouse-style scorecard for MCP servers. One command, no API key, report in seconds:
npx mcpgrade --stdio "npx -y your-mcp-server"
Then I pointed it at 36 popular servers. It did not go great.
The results
Full sortable table: https://tengli.dev/mcp-leaderboard.html. The short version (static analysis, point-in-time snapshot; servers marked (archived) are unmaintained reference implementations, included because they're still widely installed and copied):
Top of the class (A): brave-search (archived), exa, google-maps (archived), slack (archived), perplexity-ask, @shopify/dev-mcp, @apify/actors-mcp-server, airbnb, figma-developer-mcp, tavily, gitlab (archived), elastic, shrimp-task-manager, and more — 15 of 36.
Bottom of the class (D/F), 11 of 36 — and it's not hobby projects: MongoDB's official server (66, with 66 errors), Notion's official server (62), Airtable (69, 66 errors), todoist-mcp-server (67, 110 errors), GitHub's archived reference server (67, 44 errors), and firecrawl-mcp at the very bottom (57, 134 errors).
Two more servers (Stripe, Supabase) couldn't be scanned with dummy credentials and were excluded rather than graded.
Finding 1: the ecosystem has an undocumented-parameter epidemic
Almost every D/F server has a descriptions score of zero while its schema, naming, and token scores are fine. One rule dominates: D004 — parameter has no description.
firecrawl: 132 of its 134 errors are undocumented parameters. url, formats, jsonOptions — the model gets a name and a type, nothing else. todoist: 110. MongoDB and Airtable: 66 each.
The root cause is visible in the source of nearly all of them: schemas are generated from zod or OpenAPI definitions, and nobody adds .describe(). The type system knows url: string. The model needs to know which URL, in what format, with what constraints. Your schema generator is quietly stripping the single most important signal your tools have.
If you take one thing from this post: open your server, count the parameters without a description, and fix them. It's the highest-leverage hour you can spend on agent reliability.
Finding 2: it's documentation discipline, not catalog size — but size makes discipline harder
My first pass at this data suggested "small catalogs win": most 95+ scorers have few tools, and the 24–26 tool servers cluster at D/F. Then shrimp-task-manager scored A/96 with 15 tools — carefully documented, tightly named, every description distinct.
So the honest version: well-documented big catalogs are possible; they're just rare. Every tool you add is another description to write, another name that can collide, another schema to keep tight. Discipline doesn't scale by default. (Size still taxes you either way: the full catalog is serialized into every request.)
Finding 3: compliance and usability are different axes
The most-updated servers aren't the most usable ones. The archived Slack reference server — code nobody maintains — scores A/97, because someone once documented every tool and every parameter by hand. Meanwhile several actively-developed commercial servers ship parameters with no descriptions at all.
Agent usability is a writing problem more than an engineering problem. Compliance checkers can't measure it. That's the gap mcpgrade fills.
(One hopeful counterpoint: while writing this, context7 shipped a new version that fixed all its missing parameter descriptions — jumping from C to a perfect static score. The ecosystem can move fast when the gap is visible.)
Finding 4: I checked the static scores against a real model. The scary number is refusal.
Static lint is a proxy, so I built --eval: it synthesizes realistic single-step tasks (each embedding concrete values for every required parameter), shows a model the full catalog, and measures whether it picks the right tool and fills valid arguments. Calibration details and methodology: docs/eval-calibration.md. Cost: pennies per server on a small model.
Two results worth your attention:
Static findings predict live confusion. On well-documented servers, tool-selection accuracy was 100%. On firecrawl it dropped to 84% — and the misses land exactly on the naming collisions static rules flag: extract↔scrape, agent_status↔check_crawl_status, feedback↔search_feedback.
Big fuzzy catalogs break refusal. Given deliberately out-of-scope tasks, the model correctly declined 100% of the time on small, well-documented catalogs — but only 50% of the time on firecrawl's 26 fuzzy tools. Half the time it "found" a plausible tool and called it. In production, that's an agent doing something when it should do nothing — arguably the most dangerous failure mode there is.
What "good" looks like
From the top scorers, a checklist:
- Every tool description answers three questions: what it does, when to use it, what it returns.
- Every parameter has a description with format and one example value.
- Fixed value sets live in
enum, not in prose. -
requiredis declared explicitly — even when it's empty. - One naming convention, verb_object style, no generic verbs, no near-twin names.
- Errors name the missing/invalid parameter so the model can self-correct in one turn. ## Try it on your server
npx mcpgrade --stdio "node ./my-server.js" # local stdio
npx mcpgrade https://my-server.example/mcp # streamable HTTP
npx mcpgrade <target> --fail-on error # CI gate
npx mcpgrade <target> --eval # live model test (BYO key; any OpenAI-compatible endpoint works)
24 rules, each with a concrete fix and a rationale you're welcome to dispute in the issues — the ruleset is opinionated by design, and I'd rather have the argument in public. (How this differs from mcp-lint and other MCP QA tools — with side-by-side outputs: docs/comparison.md.)
If you maintain one of the servers above and fix your score, open a rescan issue — I'll happily re-run and update the table. PRs to your own servers beat arguments with my ruleset.
I build production AI agent integrations at a large tech company; mcpgrade is a personal project and reflects scars from integrating dozens of MCP connectors. No affiliation with any server ranked above.
Top comments (27)
What makes the missing .describe() finding land is that it's a generator problem, not a laziness one. Once you build schemas from zod or OpenAPI, a description just isn't a required field, so nobody notices it's gone. Would a single rule that fails on "has a type, no description" catch most of the D/F servers on its own?
Pretty much, yes — that rule exists (D004) and it dominates the failure data:
firecrawl 132 of 204 findings, todoist 76, mongodb 66. Almost every D/F server
zeroes out the descriptions category on D004 alone. If you only had budget for
one rule, that's the one.
Where the other 23 earn their keep is ranking and diagnosis rather than
detection: density normalization (132 missing params across 26 tools is a
different disease than 3 across 2), and the naming/consistency rules catch the
failures that D004 can't — the near-twin tool names that turned out to predict
live model confusion in the eval rounds. One rule finds the epidemic; the rest
tell you which patient is sickest and why.
i poked through the repo and leaderboard.html feels like the part worth dogfooding here. if a server maintainer disputes one row, opening a rescan issue works for a full rerun, but a smaller note like “this description rule is wrong for this tool” loses the exact row/version pretty fast.
i’m building PreApp around that handoff: publish the existing html from the agent, leave a note on the row, then pull the note + file/version back into the agent before regenerating. happy to run the current leaderboard through it and leave one real note if useful. no new demo file needed.
Fair observation — row-level disputes do lose their anchor today. The cheap
fix on my side: stable anchor links per leaderboard row, a scanned-version
column, and a "dispute" issue template that pre-fills server + version +
rule ID so a small note doesn't require a full rescan thread. Adding that
to the backlog.
Sure — feel free to run the published HTML through PreApp and share what
the note flow looks like. One constraint on my end: the canonical data
stays in results.ndjson in the repo (the HTML is a generated artifact),
so any annotation layer would need to round-trip through that. Curious
what it looks like regardless.
Glad we’re thinking along the same lines. One thing I’d also watch for is metric gaming. Once a score becomes a target, people naturally optimize for the score rather than real agent behavior (Goodhart’s law). That’s why I like your idea of validating against production telemetry—metrics such as repair turns, completion rate, and false-positive tool calls are much harder to game than static lint scores alone.
Reference: Goodhart’s law – “When a measure becomes a target, it ceases to be a good measure.”
en.wikipedia.org/wiki/Goodhart%27s...
I'll pass on installing the setup flow into my agent — I keep third-party
tooling out of the pipeline that generates the published artifacts, as a
rule. Nothing personal about PreApp.
But the repo is MIT and the leaderboard HTML + results.ndjson are public:
if you want to demo the round-trip, fork it, run your note flow on your
fork, and post a link or the payload here. If the locator can map a note
back to the right ndjson record cleanly, that's interesting on its own,
and I'm happy to look at the format.
Glad we’re converging on the same idea. 🙂 I also wonder whether the long-term value comes from closing the feedback loop rather than just improving the score. If anonymized production telemetry can continuously refine both the lint rules and the evaluation corpus, the benchmark becomes much harder to game and much more representative of real agent behavior. Static analysis then becomes a starting point, while production observations become the mechanism that keeps it honest.
Great write-up. One thing I’d be curious to see is whether the static score actually correlates with production behavior. Metrics like tool selection accuracy, argument repair rate, and completion rate could help validate whether a better lint score consistently leads to more reliable agents. Nice work!
Thanks! That correlation question is exactly what the calibration rounds were
for: on firecrawl, live tool-selection misses landed precisely on the
near-twin names the static rules had flagged (extract↔scrape etc.), and
refusal accuracy halved on the biggest fuzziest catalog. Static score isn't a
proxy for everything, but its worst findings do predict live confusion.
Methodology + raw numbers: github.com/TengByte/mcpgrade/blob/main/docs/eval-calibration.md
(longer write-up coming this week).
True production correlation (completion rate, argument-repair rate on real
traces) is the holy grail — that needs telemetry from a real deployment, and
I'd love to collaborate with anyone who can share anonymized traces.
The refusal result is the most useful signal here. For the eval, I'd separate four outcomes instead of a single accuracy number: correct tool and args, correct refusal, correct clarification, and unsafe/plausible action. Their costs are very different, so a server that turns ambiguity into a clarifying question should not be scored like one that silently chooses a near-twin tool.
To keep synthetic tasks from flattering the schemas that generated them, use held-out authoring: derive the intent set from real support/integration failures, have a separate process paraphrase it without seeing tool names, and freeze a test split before changing descriptions. Report confidence intervals and slices for negative tasks, overlapping names, optional-vs-required fields, enums, long catalogs, and multiple valid tools.
I'd also pin server contract, model, client prompt/tool serializer, temperature, and catalog selection policy in every result. A score is otherwise hard to compare across releases. The production correlation I would watch is repair turns per successful operation plus false-positive tool calls per 1,000 out-of-scope requests; those capture both usability cost and the dangerous “do something” failure mode.
tracked, shipped, thanks
Follow-up: your points are now tracked issues on the repo, credited to this
comment — github.com/TengByte/mcpgrade/issues
One direct ask: held-out authoring is the piece I'd most value a second
brain on. If you're at all interested in sketching the design (in the issue
thread, zero commitment), the door is open.
This is the most useful comment I've gotten anywhere on this project — thank you.
You're right on all three counts. The current eval collapses "clarifying
question" into refusal, and their costs are obviously different; a four-outcome
taxonomy (correct call / correct refusal / correct clarification / unsafe
plausible action) is strictly better and I'm stealing it. The self-flattering
risk is real too — synthesis currently sees tool names, so held-out authoring
with a paraphrase step that never sees the catalog is the right fix. And result
pinning (model, temperature, serializer, catalog policy) is only partial today:
model + temp are recorded, the rest isn't.
"Repair turns per successful operation + false-positive calls per 1k
out-of-scope requests" is a better production north star than anything I had
written down.
Would you mind if I turn this into tracked issues on the repo with credit? If
you'd rather file them yourself I'll tag them as roadmap directly.
Ran your argument against my own stack, three MCP servers I operate, ~50 tools between them. The findings sorted into three buckets, and only one is type-shaped:
Constraints that exist only in prose. Row caps, allowlisted paths, accepted time formats. Every parameter had title + type and no description. One file-read tool's allowlist was discoverable only by triggering an error and reading the message.
Descriptions that contradict the code. Two deploy tools described themselves as working; we'd known internally for weeks that they silently skip files. Spec-perfect, semantically false, on a write path. That's the one that could actually cost something.
Cross-server divergence, the bucket I didn't expect. Same tool name exposed by two servers: params was a list on one and a comma-joined string on the other, so any bind value containing a comma was silently split into two. One accepted EXPLAIN, the other rejected it. One refused to read secret-bearing files, the other served them. Nothing in either schema distinguished them, so which server the model happened to call determined the semantics.
To the categorization question upthread: the split that mattered for me wasn't type-vs-semantic, it was observable vs silent. One search tool capped matches per-file rather than in total and returned no truncation flag, so "no match" and "not searched" were the same response. Undocumented-but-loud costs a round trip; undocumented-and-silent gets believed. Worth grading separately.
One thing that may affect --eval: clients cache tool schemas. I corrected several descriptions, redeployed, and my client kept serving the old ones until it reconnected. A fixed description isn't a delivered fix, which arguably strengthens the case for catching these before they ship.
This is the first field report of someone running the argument against a
production stack, and it's worth more than the original 36-server table —
thank you.
Your three buckets map neatly onto what static analysis can and cannot see.
Bucket 1 (constraints living only in prose/error messages) is exactly D004
territory. Bucket 2 — descriptions that contradict the code — is the one
static lint can never catch: a lying description is spec-perfect by
definition. Only live probing (calling the tool and diffing claimed vs
actual behavior) gets there, and "description truthfulness on write paths"
just became the strongest argument for expanding --probe.
Bucket 3 is the one I hadn't planned for at all: cross-server semantic
collisions. Today mcpgrade grades one server at a time, so a client whose
config aggregates several servers has failure modes no single-server scan
can see. A --workspace mode that scans a full client config and flags
same-name/different-semantics collisions across servers is now on the list.
And "observable vs silent" is a better severity axis than type-vs-semantic —
undocumented-but-loud costs a round trip; undocumented-and-silent gets
believed. A silent truncation with no flag isn't a documentation bug, it's a
misinformation bug. I'm going to grade those separately, credited to this
comment.
The schema-caching point is a great practical catch too: a fixed description
isn't a delivered fix until clients reconnect — which is the whole case for
gating in CI rather than patching after ship.
If you're up for it, I'd love these as issues on the repo in your words —
but either way they're going in.
Follow-up: your findings are now tracked issues, quoted and credited —
Cross-server semantic collisions (--workspace mode, X001–X004 rule sketch):
github.com/TengByte/mcpgrade/issues/6
Silent-vs-observable failure grading for --probe (+ the lying-description
problem on write paths): github.com/TengByte/mcpgrade/issues/7
"Undocumented-but-loud costs a round trip; undocumented-and-silent gets
believed" is going to outlive this comment thread. Thanks again — if you
want to weigh in on either design, the threads are open.
This is a valuable reminder that MCP compliance and agent usability are two completely different things. We've seen cases where the protocol implementation was technically correct, but vague tool descriptions or poorly documented parameters caused agents to choose the wrong tool or hallucinate arguments anyway.
One practice that's worked well for us at IT Path Solutions is treating tool schemas as part of the agent experience, not just the API contract. Clear descriptions, explicit parameter semantics, consistent naming, and deterministic error messages usually improve reliability far more than tweaking prompts or swapping models. The refusal-rate observation was especially interesting knowing when not to call a tool is just as important as choosing the right one.
"Tool schemas as part of the agent experience, not just the API contract" is
a great way to put it — might steal that framing. And agreed on deterministic
errors: it's the most under-rated item on the list, which is why mcpgrade's
--probe mode live-calls tools and grades whether the error message names the
missing/invalid parameter. An agent that gets "invalid request" retries
blindly; one that gets "missing required field: channel_id" self-corrects in
one turn.
The insight about descriptions being the missing link between spec compliance and actual agent usability is exactly right. What I find interesting is that this problem is structural, not just developers being lazy. The zod to JSON Schema pipeline treats descriptions as optional metadata because the type system was designed for humans reading IDE tooltips, not for models making runtime decisions. A human developer can infer what url string means from the tool name alone. A model gets a flat list of parameters with no priors.
The real fix probably isnt just adding describe calls everywhere, though that is a good start. The deeper question is whether MCP tool descriptions should be first class required fields in the spec rather than optional annotations. If the protocol required descriptions for every parameter, the zod generators would have to surface them as required inputs and the ecosystem would adapt quickly. Until then, tools like mcpgrade at least make the gap visible, which is the first step toward fixing it. The fact that firecrawl alone has 132 undocumented parameters out of 134 total errors shows how widespread this is.
The structural framing is exactly right, and it's what the data shows: the
failures concentrate almost perfectly in generated schemas. Nobody hand-writes
132 empty descriptions — a pipeline does that. The type system was built for
humans with IDE tooltips and priors; models get a flat list of strings and no
tribal knowledge.
On making descriptions required in the spec: I'd love the outcome but I'm
wary of the mechanism. A hard protocol requirement breaks every existing
server on day one, and mandated descriptions have a failure mode of their
own — "url: the url" satisfies the validator and helps nobody. Quality can't
be a protocol-level constraint; presence can.
The realistic path is probably the ESLint one: norms get enforced by tooling
and distribution layers first, and specs codify them later. If registries
surfaced a usability signal and clients warned on undocumented params,
generators would add .describe() prompts within a release cycle. That said —
a spec-level SHOULD for parameter descriptions feels very proposable, and a
36-server dataset showing D004 dominance is exactly the evidence such a
proposal needs. If anyone wants to raise it in the modelcontextprotocol spec
discussions, I'll gladly contribute the data.
"Your MCP server can be 100% spec-compliant and still be unusable" — this is a brutal, universal engineering truth. Brilliant work on
mcpgradeand exposing the gap between type-safety and semantic-safety.What you're describing with the "undocumented-parameter epidemic" is essentially the AI equivalent of Primitive Obsession. Autogenerators know a variable is a
string, but they strip the semantic boundary. In your world, that missing boundary causes an LLM to hallucinate a response. In my world, it causes catastrophic network desyncs.I build strictly deterministic C++ state sync cores for the Medium-Frequency Trading (MFT) space. I recently finalized the monolithic architecture for my own engine (TolmachЁv SDK v36.0.0). When your baseline requirement is pushing 41.5M TPS with ~24ns physical RTT and absolute zero CPU validation waste, passing a raw, loosely-defined primitive across an API boundary is lethal. If an interface lacks rigid, compile-time semantic constraints, the state machine doesn't just guess wrong—it instantly corrupts the deterministic network topology.
Your insight that tools optimize for compilation rather than execution context is spot on.
Quick question regarding your
--evallive testing: When the model encounters these fuzzy, undocumented parameters and decides to hallucinate an argument, doesmcpgradecategorize the type of failure (e.g., strict type mismatch vs. semantic domain error), or is it graded as a binary pass/fail for that specific tool call?Good question, and the honest answer: partially. Argument validation today
checks schema-validity — wrong type, missing required field, out-of-enum
value — so strict mismatches are caught and attributed. What it does NOT yet
distinguish is the semantic domain error: an argument that validates cleanly
but is wrong for the task (plausible-looking URL, wrong channel id). That
distinction is part of a four-outcome taxonomy a reader proposed that's now a
tracked issue on the repo — separating "invalid args" from "valid-but-wrong
args" is exactly the kind of split that makes failure data actionable.
MCP server quality is going to matter more than people expect. A broken tool description or unsafe schema does not just fail one call; it teaches the agent the wrong affordances for the whole task.
Well put — and it compounds: a model that misreads a tool's affordance doesn't
just fail that call, it plans the rest of the task around the misreading.
Within a session, bad tool docs don't cause one error; they install a wrong
belief.
Hi,
The .describe()-skipped-because-it's-optional pattern rings very true, and I think it points at something structural: in most stacks the description is metadata bolted onto a schema that would compile fine without it, so it's the first thing that gets skipped under deadline pressure — same as comments, same as tests.
The angle I ended up taking (building an MCP-generating compiler, Archstone) was to make the description not-optional at the language level: the source you write isn't a JSON Schema with annotations, it's a business-level definition where description is a required field of the grammar itself — a capability with no description doesn't compile, full stop, same as one with no id. It doesn't fix already-deployed servers (your lint tool is exactly the right layer for that), but it does mean the D/F-grade failure mode you found — a schema that's technically valid and semantically empty — can't be produced by construction, because there's no path from source to tool definition that skips the field.
Curious whether you saw any correlation between description quality and how the schema was authored (hand-written vs. generated from an existing type/route) in the 36 you scanned — my hunch from your zod/OpenAPI examples is it's almost entirely the latter.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.