DEV Community: Teng

My eval said a perfect MCP server was broken. It was the eval that was lying.

Teng — Wed, 29 Jul 2026 00:43:20 +0000

Originally published at tengli.dev

When I added an LLM-powered eval to mcpgrade, the first real run produced a result that looked like a scoop: context7 — a server with a perfect static score — failed tool selection 62% of the time. A model shown its two-tool catalog picked the "wrong" tool on 5 of 8 tasks.

If I had shipped that number, it would have been wrong. Not slightly wrong — systematically, unfairly wrong. This post is about how I caught it, because the failure mode generalizes to most agent benchmarks people are building right now.

The setup

mcpgrade's --eval mode works like this: it reads a server's tool catalog, synthesizes realistic single-step tasks ("find the Slack channel where the incident was discussed"), shows a model the full catalog, and measures three things — does it pick the right tool, does it fill valid arguments, and does it correctly refuse tasks that no tool can handle.

Round 1, on three real servers, cost about twelve cents and produced this:

Server	Static score	Tool selection	Args	Refusal
context7 (2 tools)	100	38%	100%	100%
server-memory (9 tools)	81	93%	100%	100%
server-slack (8 tools)	97	54%	100%	100%

Two servers with excellent static scores, apparently failing live. Either static analysis was worthless, or the eval was broken.

The eval was broken

Every "miss" traced to one cause. Slack's post_message needs a thread_ts — a value you can only get from a previous call to get_channel_history. context7's get-library-docs needs a library ID that comes from resolve-library-id. These are pipelined tools: their required arguments are produced by other tools.

My task synthesizer didn't know that. It generated tasks like "reply to the thread about the outage" — without a thread timestamp. The model, quite sensibly, picked get_channel_history first (to find the thread), or declined. My grader marked both choices wrong.

The model wasn't confused. The model was right. The benchmark was grading correct multi-step reasoning as failure — and memory's 93% was the tell all along: its tools are single-step, so it scored fine.

The fix was one constraint in the synthesis prompt: every task must embed concrete values for every required parameter. "Reply to thread 1721581200.123456 in #incidents" — now single-shot selection is a fair question. Round 2: context7 38% → 100%, slack 54% → 100%.

If your agent benchmark shows a capable model failing on tools that real users navigate fine, check whether you're asking one-step questions about multi-step tools. In my experience most home-grown "tool selection accuracy" numbers have this bug quietly inflating their failure rates.

Round 3: does it discriminate?

A benchmark that gives everyone 100% is decoration. So round 3 pointed the fixed eval at firecrawl — 26 tools, the lowest static score in my 36-server scan. If the eval is measuring something real, a messy catalog should score worse. It did, in two specific ways:

1. Selection misses landed exactly on the naming collisions static rules had flagged. 84% selection accuracy, and the 16% wasn't random: extract↔scrape, agent_status↔check_crawl_status, feedback↔search_feedback — the same confusable pairs the static rules (N002, C001) had already flagged from names and descriptions alone. That's the result I most wanted: static lint predicts live model confusion. The cheap, free, ten-second scan finds the same failure points as the LLM eval.

2. Refusal collapsed. Given deliberately out-of-scope tasks, models refused correctly 100% of the time on small, well-documented catalogs — and 50% of the time on firecrawl's 26 fuzzy tools. Half the time, the model "found" a plausible-sounding tool and called it anyway. Big vague catalogs don't just cause wrong picks; they cause action when inaction is correct, which in production is the scariest failure mode there is. Nobody reviews the agent that confidently did something.

What three rounds bought me

The whole calibration — three rounds, four servers — cost about $0.60 on a small model. For that I got answers to the three questions any eval must survive:

Is it fair? After the synthesis fix, well-designed servers score 100%. Failures now mean something.
Does it discriminate? Messy catalogs score measurably worse, in interpretable ways.
Is it affordable? ~$0.04–0.2 per server. Running it on every PR is a rounding error.

Most benchmark builders skip straight to leaderboards. The calibration step — deliberately trying to prove your own metric is lying — is cheap, unglamorous, and the only thing separating a measurement from a random number generator with axes.

The calibration isn't finished — a reader proved it

Within a day of the launch post, a reader (Mads Hansen, in the dev.to comments) pointed out two flaws I hadn't caught, and he's right on both.

First: my outcome taxonomy is still too coarse. "Refusal" currently lumps together a model that asks a clarifying question and a model that declines outright — and neither is separated from the truly dangerous outcome, confidently calling a plausible-but-wrong tool. Four buckets (correct call / correct refusal / correct clarification / unsafe plausible action) with different weights is strictly better, because their production costs are wildly different.

Second, and subtler: my synthetic tasks can flatter the schemas that generated them. The synthesizer reads the catalog to write tasks — so a badly-written catalog produces tasks phrased in its own bad vocabulary. The fix is held-out authoring: derive intents from real integration failures, paraphrase them through a step that never sees tool names, and freeze the test split before touching any descriptions.

Both are now tracked issues on the repo. Which is the point of publishing your methodology instead of just your leaderboard: readers debug your benchmark the way they'd debug your code.

Full raw numbers and methodology live in the repo: docs/eval-calibration.md. If you build agent benchmarks and have found other systematic unfairness patterns, I want to hear about them — open an issue.

I build production AI agent integrations at a large tech company; mcpgrade is a personal project. The eval runs on any OpenAI-compatible endpoint — bring your own key.

I lint-scanned 36 popular MCP servers. A third of them are failing your agent.

Teng — Wed, 22 Jul 2026 06:22:03 +0000

Originally published at tengli.dev

Your MCP server can be 100% spec-compliant and still be unusable by an agent.

The Model Context Protocol spec tells you how to transport tools: JSON-RPC framing, capability negotiation, schema shapes. It says nothing about whether a model can actually use what you serve — whether it picks the right tool out of your catalog, fills the arguments correctly, or burns 8k tokens parsing your schemas on every single request.

I integrate first- and third-party MCP connectors into a production AI agent for a living, and I kept seeing the same failure: servers that pass every compliance check, yet the model calls the wrong tool, hallucinates arguments, or ignores the tool entirely. The problems were never in the protocol layer. They were in the parts no one lints: descriptions, naming, schema design.

So I wrote mcpgrade — a Lighthouse-style scorecard for MCP servers. One command, no API key, report in seconds:

npx mcpgrade --stdio "npx -y your-mcp-server"

Then I pointed it at 36 popular servers. It did not go great.

The results

Full sortable table: https://tengli.dev/mcp-leaderboard.html. The short version (static analysis, point-in-time snapshot; servers marked (archived) are unmaintained reference implementations, included because they're still widely installed and copied):

Top of the class (A): brave-search (archived), exa, google-maps (archived), slack (archived), perplexity-ask, @shopify/dev-mcp, @apify/actors-mcp-server, airbnb, figma-developer-mcp, tavily, gitlab (archived), elastic, shrimp-task-manager, and more — 15 of 36.

Bottom of the class (D/F), 11 of 36 — and it's not hobby projects: MongoDB's official server (66, with 66 errors), Notion's official server (62), Airtable (69, 66 errors), todoist-mcp-server (67, 110 errors), GitHub's archived reference server (67, 44 errors), and firecrawl-mcp at the very bottom (57, 134 errors).

Two more servers (Stripe, Supabase) couldn't be scanned with dummy credentials and were excluded rather than graded.

Finding 1: the ecosystem has an undocumented-parameter epidemic

Almost every D/F server has a descriptions score of zero while its schema, naming, and token scores are fine. One rule dominates: D004 — parameter has no description.

firecrawl: 132 of its 134 errors are undocumented parameters. url, formats, jsonOptions — the model gets a name and a type, nothing else. todoist: 110. MongoDB and Airtable: 66 each.

The root cause is visible in the source of nearly all of them: schemas are generated from zod or OpenAPI definitions, and nobody adds .describe(). The type system knows url: string. The model needs to know which URL, in what format, with what constraints. Your schema generator is quietly stripping the single most important signal your tools have.

If you take one thing from this post: open your server, count the parameters without a description, and fix them. It's the highest-leverage hour you can spend on agent reliability.

Finding 2: it's documentation discipline, not catalog size — but size makes discipline harder

My first pass at this data suggested "small catalogs win": most 95+ scorers have few tools, and the 24–26 tool servers cluster at D/F. Then shrimp-task-manager scored A/96 with 15 tools — carefully documented, tightly named, every description distinct.

So the honest version: well-documented big catalogs are possible; they're just rare. Every tool you add is another description to write, another name that can collide, another schema to keep tight. Discipline doesn't scale by default. (Size still taxes you either way: the full catalog is serialized into every request.)

Finding 3: compliance and usability are different axes

The most-updated servers aren't the most usable ones. The archived Slack reference server — code nobody maintains — scores A/97, because someone once documented every tool and every parameter by hand. Meanwhile several actively-developed commercial servers ship parameters with no descriptions at all.

Agent usability is a writing problem more than an engineering problem. Compliance checkers can't measure it. That's the gap mcpgrade fills.

(One hopeful counterpoint: while writing this, context7 shipped a new version that fixed all its missing parameter descriptions — jumping from C to a perfect static score. The ecosystem can move fast when the gap is visible.)

Finding 4: I checked the static scores against a real model. The scary number is refusal.

Static lint is a proxy, so I built --eval: it synthesizes realistic single-step tasks (each embedding concrete values for every required parameter), shows a model the full catalog, and measures whether it picks the right tool and fills valid arguments. Calibration details and methodology: docs/eval-calibration.md. Cost: pennies per server on a small model.

Two results worth your attention:

Static findings predict live confusion. On well-documented servers, tool-selection accuracy was 100%. On firecrawl it dropped to 84% — and the misses land exactly on the naming collisions static rules flag: extract↔scrape, agent_status↔check_crawl_status, feedback↔search_feedback.

Big fuzzy catalogs break refusal. Given deliberately out-of-scope tasks, the model correctly declined 100% of the time on small, well-documented catalogs — but only 50% of the time on firecrawl's 26 fuzzy tools. Half the time it "found" a plausible tool and called it. In production, that's an agent doing something when it should do nothing — arguably the most dangerous failure mode there is.

What "good" looks like

From the top scorers, a checklist:

Every tool description answers three questions: what it does, when to use it, what it returns.
Every parameter has a description with format and one example value.
Fixed value sets live in enum, not in prose.
required is declared explicitly — even when it's empty.
One naming convention, verb_object style, no generic verbs, no near-twin names.
Errors name the missing/invalid parameter so the model can self-correct in one turn. ## Try it on your server

npx mcpgrade --stdio "node ./my-server.js"   # local stdio
npx mcpgrade https://my-server.example/mcp    # streamable HTTP
npx mcpgrade <target> --fail-on error         # CI gate
npx mcpgrade <target> --eval                  # live model test (BYO key; any OpenAI-compatible endpoint works)

24 rules, each with a concrete fix and a rationale you're welcome to dispute in the issues — the ruleset is opinionated by design, and I'd rather have the argument in public. (How this differs from mcp-lint and other MCP QA tools — with side-by-side outputs: docs/comparison.md.)

If you maintain one of the servers above and fix your score, open a rescan issue — I'll happily re-run and update the table. PRs to your own servers beat arguments with my ruleset.

I build production AI agent integrations at a large tech company; mcpgrade is a personal project and reflects scars from integrating dozens of MCP connectors. No affiliation with any server ranked above.