Originally published at tengli.dev
When I added an LLM-powered eval to mcpgrade, the first real run produced a result that looked like a scoop: context7 — a server with a perfect static score — failed tool selection 62% of the time. A model shown its two-tool catalog picked the "wrong" tool on 5 of 8 tasks.
If I had shipped that number, it would have been wrong. Not slightly wrong — systematically, unfairly wrong. This post is about how I caught it, because the failure mode generalizes to most agent benchmarks people are building right now.
The setup
mcpgrade's --eval mode works like this: it reads a server's tool catalog, synthesizes realistic single-step tasks ("find the Slack channel where the incident was discussed"), shows a model the full catalog, and measures three things — does it pick the right tool, does it fill valid arguments, and does it correctly refuse tasks that no tool can handle.
Round 1, on three real servers, cost about twelve cents and produced this:
| Server | Static score | Tool selection | Args | Refusal |
|---|---|---|---|---|
| context7 (2 tools) | 100 | 38% | 100% | 100% |
| server-memory (9 tools) | 81 | 93% | 100% | 100% |
| server-slack (8 tools) | 97 | 54% | 100% | 100% |
Two servers with excellent static scores, apparently failing live. Either static analysis was worthless, or the eval was broken.
The eval was broken
Every "miss" traced to one cause. Slack's post_message needs a thread_ts — a value you can only get from a previous call to get_channel_history. context7's get-library-docs needs a library ID that comes from resolve-library-id. These are pipelined tools: their required arguments are produced by other tools.
My task synthesizer didn't know that. It generated tasks like "reply to the thread about the outage" — without a thread timestamp. The model, quite sensibly, picked get_channel_history first (to find the thread), or declined. My grader marked both choices wrong.
The model wasn't confused. The model was right. The benchmark was grading correct multi-step reasoning as failure — and memory's 93% was the tell all along: its tools are single-step, so it scored fine.
The fix was one constraint in the synthesis prompt: every task must embed concrete values for every required parameter. "Reply to thread 1721581200.123456 in #incidents" — now single-shot selection is a fair question. Round 2: context7 38% → 100%, slack 54% → 100%.
If your agent benchmark shows a capable model failing on tools that real users navigate fine, check whether you're asking one-step questions about multi-step tools. In my experience most home-grown "tool selection accuracy" numbers have this bug quietly inflating their failure rates.
Round 3: does it discriminate?
A benchmark that gives everyone 100% is decoration. So round 3 pointed the fixed eval at firecrawl — 26 tools, the lowest static score in my 36-server scan. If the eval is measuring something real, a messy catalog should score worse. It did, in two specific ways:
1. Selection misses landed exactly on the naming collisions static rules had flagged. 84% selection accuracy, and the 16% wasn't random: extract↔scrape, agent_status↔check_crawl_status, feedback↔search_feedback — the same confusable pairs the static rules (N002, C001) had already flagged from names and descriptions alone. That's the result I most wanted: static lint predicts live model confusion. The cheap, free, ten-second scan finds the same failure points as the LLM eval.
2. Refusal collapsed. Given deliberately out-of-scope tasks, models refused correctly 100% of the time on small, well-documented catalogs — and 50% of the time on firecrawl's 26 fuzzy tools. Half the time, the model "found" a plausible-sounding tool and called it anyway. Big vague catalogs don't just cause wrong picks; they cause action when inaction is correct, which in production is the scariest failure mode there is. Nobody reviews the agent that confidently did something.
What three rounds bought me
The whole calibration — three rounds, four servers — cost about $0.60 on a small model. For that I got answers to the three questions any eval must survive:
- Is it fair? After the synthesis fix, well-designed servers score 100%. Failures now mean something.
- Does it discriminate? Messy catalogs score measurably worse, in interpretable ways.
- Is it affordable? ~$0.04–0.2 per server. Running it on every PR is a rounding error.
Most benchmark builders skip straight to leaderboards. The calibration step — deliberately trying to prove your own metric is lying — is cheap, unglamorous, and the only thing separating a measurement from a random number generator with axes.
The calibration isn't finished — a reader proved it
Within a day of the launch post, a reader (Mads Hansen, in the dev.to comments) pointed out two flaws I hadn't caught, and he's right on both.
First: my outcome taxonomy is still too coarse. "Refusal" currently lumps together a model that asks a clarifying question and a model that declines outright — and neither is separated from the truly dangerous outcome, confidently calling a plausible-but-wrong tool. Four buckets (correct call / correct refusal / correct clarification / unsafe plausible action) with different weights is strictly better, because their production costs are wildly different.
Second, and subtler: my synthetic tasks can flatter the schemas that generated them. The synthesizer reads the catalog to write tasks — so a badly-written catalog produces tasks phrased in its own bad vocabulary. The fix is held-out authoring: derive intents from real integration failures, paraphrase them through a step that never sees tool names, and freeze the test split before touching any descriptions.
Both are now tracked issues on the repo. Which is the point of publishing your methodology instead of just your leaderboard: readers debug your benchmark the way they'd debug your code.
Full raw numbers and methodology live in the repo: docs/eval-calibration.md. If you build agent benchmarks and have found other systematic unfairness patterns, I want to hear about them — open an issue.
I build production AI agent integrations at a large tech company; mcpgrade is a personal project. The eval runs on any OpenAI-compatible endpoint — bring your own key.
Top comments (1)
The synthesis fix makes single-shot selection fair, but it also removes the thing that was arguably most worth measuring: whether the model works out that thread_ts has to come from get_channel_history. After the fix, a server whose tools chain badly and one whose tools chain cleanly both score 100%. Is pipelining going to get its own track, or is it out of scope for a per-server grade?