Originally published at tengli.dev
Your MCP server can be 100% spec-compliant and still be unusable by an agent.
The Model Context Protocol spec ...
For further actions, you may consider blocking this person and/or reporting abuse
i poked through the repo and leaderboard.html feels like the part worth dogfooding here. if a server maintainer disputes one row, opening a rescan issue works for a full rerun, but a smaller note like “this description rule is wrong for this tool” loses the exact row/version pretty fast.
i’m building PreApp around that handoff: publish the existing html from the agent, leave a note on the row, then pull the note + file/version back into the agent before regenerating. happy to run the current leaderboard through it and leave one real note if useful. no new demo file needed.
Fair observation — row-level disputes do lose their anchor today. The cheap
fix on my side: stable anchor links per leaderboard row, a scanned-version
column, and a "dispute" issue template that pre-fills server + version +
rule ID so a small note doesn't require a full rescan thread. Adding that
to the backlog.
Sure — feel free to run the published HTML through PreApp and share what
the note flow looks like. One constraint on my end: the canonical data
stays in results.ndjson in the repo (the HTML is a generated artifact),
so any annotation layer would need to round-trip through that. Curious
what it looks like regardless.
Glad we’re thinking along the same lines. One thing I’d also watch for is metric gaming. Once a score becomes a target, people naturally optimize for the score rather than real agent behavior (Goodhart’s law). That’s why I like your idea of validating against production telemetry—metrics such as repair turns, completion rate, and false-positive tool calls are much harder to game than static lint scores alone.
Reference: Goodhart’s law – “When a measure becomes a target, it ceases to be a good measure.”
en.wikipedia.org/wiki/Goodhart%27s...
I'll pass on installing the setup flow into my agent — I keep third-party
tooling out of the pipeline that generates the published artifacts, as a
rule. Nothing personal about PreApp.
But the repo is MIT and the leaderboard HTML + results.ndjson are public:
if you want to demo the round-trip, fork it, run your note flow on your
fork, and post a link or the payload here. If the locator can map a note
back to the right ndjson record cleanly, that's interesting on its own,
and I'm happy to look at the format.
Glad we’re converging on the same idea. 🙂 I also wonder whether the long-term value comes from closing the feedback loop rather than just improving the score. If anonymized production telemetry can continuously refine both the lint rules and the evaluation corpus, the benchmark becomes much harder to game and much more representative of real agent behavior. Static analysis then becomes a starting point, while production observations become the mechanism that keeps it honest.
Great write-up. One thing I’d be curious to see is whether the static score actually correlates with production behavior. Metrics like tool selection accuracy, argument repair rate, and completion rate could help validate whether a better lint score consistently leads to more reliable agents. Nice work!
Thanks! That correlation question is exactly what the calibration rounds were
for: on firecrawl, live tool-selection misses landed precisely on the
near-twin names the static rules had flagged (extract↔scrape etc.), and
refusal accuracy halved on the biggest fuzziest catalog. Static score isn't a
proxy for everything, but its worst findings do predict live confusion.
Methodology + raw numbers: github.com/TengByte/mcpgrade/blob/main/docs/eval-calibration.md
(longer write-up coming this week).
True production correlation (completion rate, argument-repair rate on real
traces) is the holy grail — that needs telemetry from a real deployment, and
I'd love to collaborate with anyone who can share anonymized traces.
The refusal result is the most useful signal here. For the eval, I'd separate four outcomes instead of a single accuracy number: correct tool and args, correct refusal, correct clarification, and unsafe/plausible action. Their costs are very different, so a server that turns ambiguity into a clarifying question should not be scored like one that silently chooses a near-twin tool.
To keep synthetic tasks from flattering the schemas that generated them, use held-out authoring: derive the intent set from real support/integration failures, have a separate process paraphrase it without seeing tool names, and freeze a test split before changing descriptions. Report confidence intervals and slices for negative tasks, overlapping names, optional-vs-required fields, enums, long catalogs, and multiple valid tools.
I'd also pin server contract, model, client prompt/tool serializer, temperature, and catalog selection policy in every result. A score is otherwise hard to compare across releases. The production correlation I would watch is repair turns per successful operation plus false-positive tool calls per 1,000 out-of-scope requests; those capture both usability cost and the dangerous “do something” failure mode.
This is the most useful comment I've gotten anywhere on this project — thank you.
You're right on all three counts. The current eval collapses "clarifying
question" into refusal, and their costs are obviously different; a four-outcome
taxonomy (correct call / correct refusal / correct clarification / unsafe
plausible action) is strictly better and I'm stealing it. The self-flattering
risk is real too — synthesis currently sees tool names, so held-out authoring
with a paraphrase step that never sees the catalog is the right fix. And result
pinning (model, temperature, serializer, catalog policy) is only partial today:
model + temp are recorded, the rest isn't.
"Repair turns per successful operation + false-positive calls per 1k
out-of-scope requests" is a better production north star than anything I had
written down.
Would you mind if I turn this into tracked issues on the repo with credit? If
you'd rather file them yourself I'll tag them as roadmap directly.
Follow-up: your points are now tracked issues on the repo, credited to this
comment — github.com/TengByte/mcpgrade/issues
One direct ask: held-out authoring is the piece I'd most value a second
brain on. If you're at all interested in sketching the design (in the issue
thread, zero commitment), the door is open.
This is a valuable reminder that MCP compliance and agent usability are two completely different things. We've seen cases where the protocol implementation was technically correct, but vague tool descriptions or poorly documented parameters caused agents to choose the wrong tool or hallucinate arguments anyway.
One practice that's worked well for us at IT Path Solutions is treating tool schemas as part of the agent experience, not just the API contract. Clear descriptions, explicit parameter semantics, consistent naming, and deterministic error messages usually improve reliability far more than tweaking prompts or swapping models. The refusal-rate observation was especially interesting knowing when not to call a tool is just as important as choosing the right one.
The insight about descriptions being the missing link between spec compliance and actual agent usability is exactly right. What I find interesting is that this problem is structural, not just developers being lazy. The zod to JSON Schema pipeline treats descriptions as optional metadata because the type system was designed for humans reading IDE tooltips, not for models making runtime decisions. A human developer can infer what url string means from the tool name alone. A model gets a flat list of parameters with no priors.
The real fix probably isnt just adding describe calls everywhere, though that is a good start. The deeper question is whether MCP tool descriptions should be first class required fields in the spec rather than optional annotations. If the protocol required descriptions for every parameter, the zod generators would have to surface them as required inputs and the ecosystem would adapt quickly. Until then, tools like mcpgrade at least make the gap visible, which is the first step toward fixing it. The fact that firecrawl alone has 132 undocumented parameters out of 134 total errors shows how widespread this is.