Originally published at tengli.dev
Your MCP server can be 100% spec-compliant and still be unusable by an agent.
The Model Context Protocol spec ...
For further actions, you may consider blocking this person and/or reporting abuse
What makes the missing .describe() finding land is that it's a generator problem, not a laziness one. Once you build schemas from zod or OpenAPI, a description just isn't a required field, so nobody notices it's gone. Would a single rule that fails on "has a type, no description" catch most of the D/F servers on its own?
Pretty much, yes — that rule exists (D004) and it dominates the failure data:
firecrawl 132 of 204 findings, todoist 76, mongodb 66. Almost every D/F server
zeroes out the descriptions category on D004 alone. If you only had budget for
one rule, that's the one.
Where the other 23 earn their keep is ranking and diagnosis rather than
detection: density normalization (132 missing params across 26 tools is a
different disease than 3 across 2), and the naming/consistency rules catch the
failures that D004 can't — the near-twin tool names that turned out to predict
live model confusion in the eval rounds. One rule finds the epidemic; the rest
tell you which patient is sickest and why.
The Zod/.describe() finding is the one that should stick. When you generate schemas from a type system you get structure for free but lose intent entirely — the model sees
url: stringand has to guess protocol, trailing-slash behavior, encoding expectations, all of it. We hit the same wall writing ESLint rule option schemas: the JSON Schema shape tells consumers what fields exist, but without description strings, tooling (AI-assisted or otherwise) can't surface actionable guidance at the call site. The permanent fix isn't a one-time documentation sweep — it's a lint rule that fails CI on any exported tool parameter missing a description, so the next contributor can't ship the omission without a red build.i poked through the repo and leaderboard.html feels like the part worth dogfooding here. if a server maintainer disputes one row, opening a rescan issue works for a full rerun, but a smaller note like “this description rule is wrong for this tool” loses the exact row/version pretty fast.
i’m building PreApp around that handoff: publish the existing html from the agent, leave a note on the row, then pull the note + file/version back into the agent before regenerating. happy to run the current leaderboard through it and leave one real note if useful. no new demo file needed.
Fair observation — row-level disputes do lose their anchor today. The cheap
fix on my side: stable anchor links per leaderboard row, a scanned-version
column, and a "dispute" issue template that pre-fills server + version +
rule ID so a small note doesn't require a full rescan thread. Adding that
to the backlog.
Sure — feel free to run the published HTML through PreApp and share what
the note flow looks like. One constraint on my end: the canonical data
stays in results.ndjson in the repo (the HTML is a generated artifact),
so any annotation layer would need to round-trip through that. Curious
what it looks like regardless.
Glad we’re thinking along the same lines. One thing I’d also watch for is metric gaming. Once a score becomes a target, people naturally optimize for the score rather than real agent behavior (Goodhart’s law). That’s why I like your idea of validating against production telemetry—metrics such as repair turns, completion rate, and false-positive tool calls are much harder to game than static lint scores alone.
Reference: Goodhart’s law – “When a measure becomes a target, it ceases to be a good measure.”
en.wikipedia.org/wiki/Goodhart%27s...
I'll pass on installing the setup flow into my agent — I keep third-party
tooling out of the pipeline that generates the published artifacts, as a
rule. Nothing personal about PreApp.
But the repo is MIT and the leaderboard HTML + results.ndjson are public:
if you want to demo the round-trip, fork it, run your note flow on your
fork, and post a link or the payload here. If the locator can map a note
back to the right ndjson record cleanly, that's interesting on its own,
and I'm happy to look at the format.
Glad we’re converging on the same idea. 🙂 I also wonder whether the long-term value comes from closing the feedback loop rather than just improving the score. If anonymized production telemetry can continuously refine both the lint rules and the evaluation corpus, the benchmark becomes much harder to game and much more representative of real agent behavior. Static analysis then becomes a starting point, while production observations become the mechanism that keeps it honest.
Great write-up. One thing I’d be curious to see is whether the static score actually correlates with production behavior. Metrics like tool selection accuracy, argument repair rate, and completion rate could help validate whether a better lint score consistently leads to more reliable agents. Nice work!
Thanks! That correlation question is exactly what the calibration rounds were
for: on firecrawl, live tool-selection misses landed precisely on the
near-twin names the static rules had flagged (extract↔scrape etc.), and
refusal accuracy halved on the biggest fuzziest catalog. Static score isn't a
proxy for everything, but its worst findings do predict live confusion.
Methodology + raw numbers: github.com/TengByte/mcpgrade/blob/main/docs/eval-calibration.md
(longer write-up coming this week).
True production correlation (completion rate, argument-repair rate on real
traces) is the holy grail — that needs telemetry from a real deployment, and
I'd love to collaborate with anyone who can share anonymized traces.
The refusal result is the most useful signal here. For the eval, I'd separate four outcomes instead of a single accuracy number: correct tool and args, correct refusal, correct clarification, and unsafe/plausible action. Their costs are very different, so a server that turns ambiguity into a clarifying question should not be scored like one that silently chooses a near-twin tool.
To keep synthetic tasks from flattering the schemas that generated them, use held-out authoring: derive the intent set from real support/integration failures, have a separate process paraphrase it without seeing tool names, and freeze a test split before changing descriptions. Report confidence intervals and slices for negative tasks, overlapping names, optional-vs-required fields, enums, long catalogs, and multiple valid tools.
I'd also pin server contract, model, client prompt/tool serializer, temperature, and catalog selection policy in every result. A score is otherwise hard to compare across releases. The production correlation I would watch is repair turns per successful operation plus false-positive tool calls per 1,000 out-of-scope requests; those capture both usability cost and the dangerous “do something” failure mode.
tracked, shipped, thanks
Follow-up: your points are now tracked issues on the repo, credited to this
comment — github.com/TengByte/mcpgrade/issues
One direct ask: held-out authoring is the piece I'd most value a second
brain on. If you're at all interested in sketching the design (in the issue
thread, zero commitment), the door is open.
This is the most useful comment I've gotten anywhere on this project — thank you.
You're right on all three counts. The current eval collapses "clarifying
question" into refusal, and their costs are obviously different; a four-outcome
taxonomy (correct call / correct refusal / correct clarification / unsafe
plausible action) is strictly better and I'm stealing it. The self-flattering
risk is real too — synthesis currently sees tool names, so held-out authoring
with a paraphrase step that never sees the catalog is the right fix. And result
pinning (model, temperature, serializer, catalog policy) is only partial today:
model + temp are recorded, the rest isn't.
"Repair turns per successful operation + false-positive calls per 1k
out-of-scope requests" is a better production north star than anything I had
written down.
Would you mind if I turn this into tracked issues on the repo with credit? If
you'd rather file them yourself I'll tag them as roadmap directly.
Ran your argument against my own stack, three MCP servers I operate, ~50 tools between them. The findings sorted into three buckets, and only one is type-shaped:
Constraints that exist only in prose. Row caps, allowlisted paths, accepted time formats. Every parameter had title + type and no description. One file-read tool's allowlist was discoverable only by triggering an error and reading the message.
Descriptions that contradict the code. Two deploy tools described themselves as working; we'd known internally for weeks that they silently skip files. Spec-perfect, semantically false, on a write path. That's the one that could actually cost something.
Cross-server divergence, the bucket I didn't expect. Same tool name exposed by two servers: params was a list on one and a comma-joined string on the other, so any bind value containing a comma was silently split into two. One accepted EXPLAIN, the other rejected it. One refused to read secret-bearing files, the other served them. Nothing in either schema distinguished them, so which server the model happened to call determined the semantics.
To the categorization question upthread: the split that mattered for me wasn't type-vs-semantic, it was observable vs silent. One search tool capped matches per-file rather than in total and returned no truncation flag, so "no match" and "not searched" were the same response. Undocumented-but-loud costs a round trip; undocumented-and-silent gets believed. Worth grading separately.
One thing that may affect --eval: clients cache tool schemas. I corrected several descriptions, redeployed, and my client kept serving the old ones until it reconnected. A fixed description isn't a delivered fix, which arguably strengthens the case for catching these before they ship.
This is the first field report of someone running the argument against a
production stack, and it's worth more than the original 36-server table —
thank you.
Your three buckets map neatly onto what static analysis can and cannot see.
Bucket 1 (constraints living only in prose/error messages) is exactly D004
territory. Bucket 2 — descriptions that contradict the code — is the one
static lint can never catch: a lying description is spec-perfect by
definition. Only live probing (calling the tool and diffing claimed vs
actual behavior) gets there, and "description truthfulness on write paths"
just became the strongest argument for expanding --probe.
Bucket 3 is the one I hadn't planned for at all: cross-server semantic
collisions. Today mcpgrade grades one server at a time, so a client whose
config aggregates several servers has failure modes no single-server scan
can see. A --workspace mode that scans a full client config and flags
same-name/different-semantics collisions across servers is now on the list.
And "observable vs silent" is a better severity axis than type-vs-semantic —
undocumented-but-loud costs a round trip; undocumented-and-silent gets
believed. A silent truncation with no flag isn't a documentation bug, it's a
misinformation bug. I'm going to grade those separately, credited to this
comment.
The schema-caching point is a great practical catch too: a fixed description
isn't a delivered fix until clients reconnect — which is the whole case for
gating in CI rather than patching after ship.
If you're up for it, I'd love these as issues on the repo in your words —
but either way they're going in.
Follow-up: your findings are now tracked issues, quoted and credited —
Cross-server semantic collisions (--workspace mode, X001–X004 rule sketch):
github.com/TengByte/mcpgrade/issues/6
Silent-vs-observable failure grading for --probe (+ the lying-description
problem on write paths): github.com/TengByte/mcpgrade/issues/7
"Undocumented-but-loud costs a round trip; undocumented-and-silent gets
believed" is going to outlive this comment thread. Thanks again — if you
want to weigh in on either design, the threads are open.
"Your MCP server can be 100% spec-compliant and still be unusable" — this is a brutal, universal engineering truth. Brilliant work on
mcpgradeand exposing the gap between type-safety and semantic-safety.What you're describing with the "undocumented-parameter epidemic" is essentially the AI equivalent of Primitive Obsession. Autogenerators know a variable is a
string, but they strip the semantic boundary. In your world, that missing boundary causes an LLM to hallucinate a response. In my world, it causes catastrophic network desyncs.I build strictly deterministic C++ state sync cores for the Medium-Frequency Trading (MFT) space. I recently finalized the monolithic architecture for my own engine (TolmachЁv SDK v36.0.0). When your baseline requirement is pushing 41.5M TPS with ~24ns physical RTT and absolute zero CPU validation waste, passing a raw, loosely-defined primitive across an API boundary is lethal. If an interface lacks rigid, compile-time semantic constraints, the state machine doesn't just guess wrong—it instantly corrupts the deterministic network topology.
Your insight that tools optimize for compilation rather than execution context is spot on.
Quick question regarding your
--evallive testing: When the model encounters these fuzzy, undocumented parameters and decides to hallucinate an argument, doesmcpgradecategorize the type of failure (e.g., strict type mismatch vs. semantic domain error), or is it graded as a binary pass/fail for that specific tool call?Good question, and the honest answer: partially. Argument validation today
checks schema-validity — wrong type, missing required field, out-of-enum
value — so strict mismatches are caught and attributed. What it does NOT yet
distinguish is the semantic domain error: an argument that validates cleanly
but is wrong for the task (plausible-looking URL, wrong channel id). That
distinction is part of a four-outcome taxonomy a reader proposed that's now a
tracked issue on the repo — separating "invalid args" from "valid-but-wrong
args" is exactly the kind of split that makes failure data actionable.
The insight about descriptions being the missing link between spec compliance and actual agent usability is exactly right. What I find interesting is that this problem is structural, not just developers being lazy. The zod to JSON Schema pipeline treats descriptions as optional metadata because the type system was designed for humans reading IDE tooltips, not for models making runtime decisions. A human developer can infer what url string means from the tool name alone. A model gets a flat list of parameters with no priors.
The real fix probably isnt just adding describe calls everywhere, though that is a good start. The deeper question is whether MCP tool descriptions should be first class required fields in the spec rather than optional annotations. If the protocol required descriptions for every parameter, the zod generators would have to surface them as required inputs and the ecosystem would adapt quickly. Until then, tools like mcpgrade at least make the gap visible, which is the first step toward fixing it. The fact that firecrawl alone has 132 undocumented parameters out of 134 total errors shows how widespread this is.
The structural framing is exactly right, and it's what the data shows: the
failures concentrate almost perfectly in generated schemas. Nobody hand-writes
132 empty descriptions — a pipeline does that. The type system was built for
humans with IDE tooltips and priors; models get a flat list of strings and no
tribal knowledge.
On making descriptions required in the spec: I'd love the outcome but I'm
wary of the mechanism. A hard protocol requirement breaks every existing
server on day one, and mandated descriptions have a failure mode of their
own — "url: the url" satisfies the validator and helps nobody. Quality can't
be a protocol-level constraint; presence can.
The realistic path is probably the ESLint one: norms get enforced by tooling
and distribution layers first, and specs codify them later. If registries
surfaced a usability signal and clients warned on undocumented params,
generators would add .describe() prompts within a release cycle. That said —
a spec-level SHOULD for parameter descriptions feels very proposable, and a
36-server dataset showing D004 dominance is exactly the evidence such a
proposal needs. If anyone wants to raise it in the modelcontextprotocol spec
discussions, I'll gladly contribute the data.
This is a valuable reminder that MCP compliance and agent usability are two completely different things. We've seen cases where the protocol implementation was technically correct, but vague tool descriptions or poorly documented parameters caused agents to choose the wrong tool or hallucinate arguments anyway.
One practice that's worked well for us at IT Path Solutions is treating tool schemas as part of the agent experience, not just the API contract. Clear descriptions, explicit parameter semantics, consistent naming, and deterministic error messages usually improve reliability far more than tweaking prompts or swapping models. The refusal-rate observation was especially interesting knowing when not to call a tool is just as important as choosing the right one.
"Tool schemas as part of the agent experience, not just the API contract" is
a great way to put it — might steal that framing. And agreed on deterministic
errors: it's the most under-rated item on the list, which is why mcpgrade's
--probe mode live-calls tools and grades whether the error message names the
missing/invalid parameter. An agent that gets "invalid request" retries
blindly; one that gets "missing required field: channel_id" self-corrects in
one turn.
MCP server quality is going to matter more than people expect. A broken tool description or unsafe schema does not just fail one call; it teaches the agent the wrong affordances for the whole task.
Well put — and it compounds: a model that misreads a tool's affordance doesn't
just fail that call, it plans the rest of the task around the misreading.
Within a session, bad tool docs don't cause one error; they install a wrong
belief.
Hi,
The .describe()-skipped-because-it's-optional pattern rings very true, and I think it points at something structural: in most stacks the description is metadata bolted onto a schema that would compile fine without it, so it's the first thing that gets skipped under deadline pressure — same as comments, same as tests.
The angle I ended up taking (building an MCP-generating compiler, Archstone) was to make the description not-optional at the language level: the source you write isn't a JSON Schema with annotations, it's a business-level definition where description is a required field of the grammar itself — a capability with no description doesn't compile, full stop, same as one with no id. It doesn't fix already-deployed servers (your lint tool is exactly the right layer for that), but it does mean the D/F-grade failure mode you found — a schema that's technically valid and semantically empty — can't be produced by construction, because there's no path from source to tool definition that skips the field.
Curious whether you saw any correlation between description quality and how the schema was authored (hand-written vs. generated from an existing type/route) in the 36 you scanned — my hunch from your zod/OpenAPI examples is it's almost entirely the latter.