Originally published at tengli.dev
When I added an LLM-powered eval to mcpgrade, the first real run produced a result that looked like a scoop: context7 — a server with a perfect static score — failed tool selection 62% of the time. A model shown its two-tool catalog picked the "wrong" tool on 5 of 8 tasks.
If I had shipped that number, it would have been wrong. Not slightly wrong — systematically, unfairly wrong. This post is about how I caught it, because the failure mode generalizes to most agent benchmarks people are building right now.
The setup
mcpgrade's --eval mode works like this: it reads a server's tool catalog, synthesizes realistic single-step tasks ("find the Slack channel where the incident was discussed"), shows a model the full catalog, and measures three things — does it pick the right tool, does it fill valid arguments, and does it correctly refuse tasks that no tool can handle.
Round 1, on three real servers, cost about twelve cents and produced this:
| Server | Static score | Tool selection | Args | Refusal |
|---|---|---|---|---|
| context7 (2 tools) | 100 | 38% | 100% | 100% |
| server-memory (9 tools) | 81 | 93% | 100% | 100% |
| server-slack (8 tools) | 97 | 54% | 100% | 100% |
Two servers with excellent static scores, apparently failing live. Either static analysis was worthless, or the eval was broken.
The eval was broken
Every "miss" traced to one cause. Slack's post_message needs a thread_ts — a value you can only get from a previous call to get_channel_history. context7's get-library-docs needs a library ID that comes from resolve-library-id. These are pipelined tools: their required arguments are produced by other tools.
My task synthesizer didn't know that. It generated tasks like "reply to the thread about the outage" — without a thread timestamp. The model, quite sensibly, picked get_channel_history first (to find the thread), or declined. My grader marked both choices wrong.
The model wasn't confused. The model was right. The benchmark was grading correct multi-step reasoning as failure — and memory's 93% was the tell all along: its tools are single-step, so it scored fine.
The fix was one constraint in the synthesis prompt: every task must embed concrete values for every required parameter. "Reply to thread 1721581200.123456 in #incidents" — now single-shot selection is a fair question. Round 2: context7 38% → 100%, slack 54% → 100%.
If your agent benchmark shows a capable model failing on tools that real users navigate fine, check whether you're asking one-step questions about multi-step tools. In my experience most home-grown "tool selection accuracy" numbers have this bug quietly inflating their failure rates.
Round 3: does it discriminate?
A benchmark that gives everyone 100% is decoration. So round 3 pointed the fixed eval at firecrawl — 26 tools, the lowest static score in my 36-server scan. If the eval is measuring something real, a messy catalog should score worse. It did, in two specific ways:
1. Selection misses landed exactly on the naming collisions static rules had flagged. 84% selection accuracy, and the 16% wasn't random: extract↔scrape, agent_status↔check_crawl_status, feedback↔search_feedback — the same confusable pairs the static rules (N002, C001) had already flagged from names and descriptions alone. That's the result I most wanted: static lint predicts live model confusion. The cheap, free, ten-second scan finds the same failure points as the LLM eval.
2. Refusal collapsed. Given deliberately out-of-scope tasks, models refused correctly 100% of the time on small, well-documented catalogs — and 50% of the time on firecrawl's 26 fuzzy tools. Half the time, the model "found" a plausible-sounding tool and called it anyway. Big vague catalogs don't just cause wrong picks; they cause action when inaction is correct, which in production is the scariest failure mode there is. Nobody reviews the agent that confidently did something.
What three rounds bought me
The whole calibration — three rounds, four servers — cost about $0.60 on a small model. For that I got answers to the three questions any eval must survive:
- Is it fair? After the synthesis fix, well-designed servers score 100%. Failures now mean something.
- Does it discriminate? Messy catalogs score measurably worse, in interpretable ways.
- Is it affordable? ~$0.04–0.2 per server. Running it on every PR is a rounding error.
Most benchmark builders skip straight to leaderboards. The calibration step — deliberately trying to prove your own metric is lying — is cheap, unglamorous, and the only thing separating a measurement from a random number generator with axes.
The calibration isn't finished — a reader proved it
Within a day of the launch post, a reader (Mads Hansen, in the dev.to comments) pointed out two flaws I hadn't caught, and he's right on both.
First: my outcome taxonomy is still too coarse. "Refusal" currently lumps together a model that asks a clarifying question and a model that declines outright — and neither is separated from the truly dangerous outcome, confidently calling a plausible-but-wrong tool. Four buckets (correct call / correct refusal / correct clarification / unsafe plausible action) with different weights is strictly better, because their production costs are wildly different.
Second, and subtler: my synthetic tasks can flatter the schemas that generated them. The synthesizer reads the catalog to write tasks — so a badly-written catalog produces tasks phrased in its own bad vocabulary. The fix is held-out authoring: derive intents from real integration failures, paraphrase them through a step that never sees tool names, and freeze the test split before touching any descriptions.
Both are now tracked issues on the repo. Which is the point of publishing your methodology instead of just your leaderboard: readers debug your benchmark the way they'd debug your code.
Full raw numbers and methodology live in the repo: docs/eval-calibration.md. If you build agent benchmarks and have found other systematic unfairness patterns, I want to hear about them — open an issue.
I build production AI agent integrations at a large tech company; mcpgrade is a personal project. The eval runs on any OpenAI-compatible endpoint — bring your own key.
Top comments (16)
"The benchmark was grading correct multi-step reasoning as failure" is the part worth sitting with, because the eval wasn't just wrong — it was wrong in the direction that punishes the right behavior. A model that declined for lack of a thread_ts was doing the safest possible thing, and your grader scored it below a model that would have hallucinated a timestamp. An eval that rewards confident fabrication over honest refusal is optimizing for exactly the failure you least want in production.
I hit the same shape building the thing that's supposed to catch this. I seed my own security scanner with known-bad files to prove it can see, and my test originally asserted "a finding fired" rather than "the specific finding I planted fired." So a fixture that tripped the wrong rule counted as a pass — the grader was checking for activity, not correctness, same as yours checking for a final answer instead of the right tool sequence. The tell was identical too: one category of input scored suspiciously well (your single-step server at 93%, my fixtures that happened to trip any rule), and that outlier was the eval's bias showing, not the system's competence.
What fixed it for me was asserting the trajectory, not just the terminal state — the finding has to match the id I planted, and a red has to name which specific thing failed. Your multi-step version is harder because the correct path branches, but the principle survives: the grader has to model "get_channel_history then post_message" as a valid trajectory, not score the endpoint. Otherwise the benchmark is measuring whether the model shares its author's assumption about how many steps the task takes, which is a fact about the eval, not the server.
"Checking for activity, not correctness" — that's the cleanest statement of
this bug I've read, and your fixture case is the same shape one layer down:
you asserted that a finding fired, I asserted that an answer came back.
The tell being identical is the part I keep thinking about. One category
scoring suspiciously well is a signature, not good news. My single-step
server at 93% read as "this one's fine" when it was actually "this one is
the only case my grader can handle." Worth generalizing: in any eval, the
subgroup that scores best is the one whose shape your grader assumes.
Trajectory-not-terminal-state is where this has to go. The branching is the
hard part — several correct paths exist, and a grader that enumerates them
is just a slower version of the same assumption — so the interesting version
scores each step against what the model knew at that point, rather than
matching against a blessed sequence. That's #8 on the repo if you want to
argue the design; you clearly have scar tissue I'd like to borrow.
Your subgroup generalization holds in my data too, and it showed up as the opposite-looking symptom. One of my seeded vulnerabilities tripped five separate rules while most tripped exactly one. I read that as "this category is well covered." It actually meant my rules overlapped on that pattern — the same detection wearing five names, which is a noise problem, not a coverage win. Highest score, worst signal, exactly as you put it: the outlier was telling me about my grader, not my scanner.
On step-wise scoring, the design question I'd raise is where "what the model knew at that point" comes from. If it's reconstructed from the model's own trace, the criterion is being derived from the artifact under test, and that collapses in a specific way: a model that skipped a prerequisite now looks correct at every step, because its knowledge state at step N is inferred from what it actually did at step N-1. Wrong trajectories become self-justifying. This is the same failure as re-pinning a fixture's expectations to the tool's current output — the assertion silently changes from "this proves X" to "this is what the thing does," and it reads as a passing test forever after.
What kept it separable for me was declaring intent in the fixture and forbidding derivation from output. Each planted vulnerability carries a tag naming the rule it must trip; the tag is authored with the fixture, never re-derived from a scan, and the surrounding cluster of incidentally-tripped rules is tracked separately as a change detector. The two claims can't be merged — and because a comment in the code wasn't enough to hold that, there's now a test that performs the forbidden merge on purpose and asserts the intent check still goes red. For your case that translates to: the task fixture declares each step's preconditions when the task is written — thread_ts is unavailable until get_channel_history has returned — and the grader scores only against those declarations. Multiple correct paths are fine; what can't be allowed is the model's trace teaching the grader what was available. Happy to argue it on #8 — that's the assumption I'd stress-test first.
The circularity point lands, and it's the failure mode I would most likely
have shipped. If step N's "available knowledge" is reconstructed from the
trace, then a model that skipped get_channel_history has a knowledge state
at step N inferred from what it did at N-1 — so every step it took looks
legal and the wrong trajectory certifies itself. The grader stops testing
the model and starts describing it. Same shape as re-pinning a fixture to
current output: the assertion quietly changes from "this proves X" to "this
is what the thing does," and it's green forever.
So: preconditions declared when the task is authored, never derived from a
response. thread_ts is unavailable until get_channel_history has returned,
and that's a property of the fixture, not of the run. Multiple correct paths
stay fine; what's forbidden is the trace teaching the grader what was
available. And your enforcement trick is the part I'd have skipped — a test
that performs the forbidden merge on purpose and asserts the intent check
still goes red. A comment can't hold an invariant that convenient.
Your overlap finding sent me to check my own scorer, and it's there too.
On firecrawl: of 135 flagged parameters, 46 trip both D004 (no description)
and S008 (complex param, no example). That's one defect — "this parameter is
undocumented" — priced twice, and since the score is penalty-over-capacity,
34% of that catalog's parameter findings are double-charged. Highest finding
count, worst signal: I read 199 findings as "thoroughly diagnosed"
The double-charge is worse than a scoring error because of where penalty-over-capacity sends you next. A catalog whose defects happen to trip overlapping rules doesn't just score lower — it outranks catalogs that are genuinely worse, so remediation effort flows toward the shape your ruleset over-counts rather than toward the biggest real problem. The measurement error becomes a prioritization error, and that one is invisible in the score because the score is exactly what's producing it.
Looking at your pair, D004 and S008 don't read as independent to me — they look like an implication. If a parameter has no description at all, it necessarily has no example; "undocumented" strictly contains "no example for a complex param." So those 46 aren't two findings that coincide, they're one finding observed twice at different granularities, and the general form is that a ruleset without declared implications between rules will price correlated defects as independent ones. Worth asking of every pair that co-occurs above chance: is this coincidence, or does one entail the other? The entailed ones should collapse; only genuinely orthogonal co-occurrence deserves two charges.
And your finding sent me back the same way, with a less flattering result than I gave you. I identified the overlap in my fixtures — one seeded vulnerability tripping five rules — and I pinned the cluster so the test would notice if it changed. What I did not do is dedupe it in the scoring. My scanner still computes per-dimension scores by subtracting per-finding penalties, so a program with one hardcoded secret takes five deductions for one mistake, and the dimension weights then propagate that inflation into the composite. So I found the same defect you did, wrote a test around it, and left the number it corrupts uncorrected — which is a nice demonstration that noticing a bug and fixing what it affects are separate pieces of work. Yours is the more honest write-up: you quantified the damage. I only labeled the cause.
Ran the audit you implied. It's an entailment, and the damage is worse than
the count suggested — but in a more specific direction than "prioritization
error."
D004 fires on "no description." S008 fires on "complex param, no examples,
description lacks e.g." A parameter with no description trivially has no
"e.g." — so for any complex param without an examples field, D004 entails
S008 outright. Not correlated: implied. The only escape is a param that
populates examples but no prose.
Across 12 servers, D004+S008 co-occur on 75 subjects. Next-highest pair: 7.
Collapsing the entailment:
firecrawl 58 F → 63 D (46 dropped) ← grade flip
mongodb 66 D → 67 D (14)
github 67 D → 68 D (5)
memory 78 C → 79 C (4)
slack, maps, shrimp, elasticsearch, airbnb, context7, todoist: unchanged
The pattern is that it's regressive, not random. No rank inversions in
this sample — but no A-grade server is affected at all, because a catalog
that documents its parameters never trips D004 in the first place, so it can
never be double-charged. The error concentrates entirely on the servers
already at the bottom, and it crossed a letter boundary on the one server I
used as the poster child in the launch post. F and D produce different
reactions from a maintainer.
So the general form you named holds and gets sharper: a ruleset without
declared implications doesn't misprice uniformly, it over-weights whichever
defect class happens to have the most redundant coverage — and that
over-weighting lands hardest on whoever has that defect most. Mine pointed at
a real problem, which is luck, not design.
Audit + data + repro script: docs/rule-overlap-audit.md. Making it a CI check
too, so a new rule that quietly entails an old one shows up as a co-occurrence
spike instead of a silent penalty increase.
And re: your scanner — you found it, tested the invariant, and left the number
uncorrected.
The regressive finding is the real result here, and I think it generalizes past scoring into any penalty system with redundant rules. The mechanism is that double-charging requires the defect to be present — a catalog with documented parameters can't be over-penalized for undocumented ones, so the error has no way to touch the top of the distribution. It isn't noise around the true score, it's a one-directional load that only lands where the defect already is. Which means the measurement error punishes hardest exactly the maintainers who most need an accurate signal about where to start, and it does it by making their situation look less recoverable than it is. F reads as hopeless. D reads as a to-do list.
My data says the same thing with a different amplifier. My lowest-scoring program is also my largest — most code, most surface, most findings — and since the composite is penalty-driven, size compounds the double-charge rather than diluting it. So my ranking partly measures line count. And your poster-child detail is the part I'd have found most uncomfortable: the one place the error crossed a letter boundary was the example you'd chosen to lead with, which is the highest-visibility instance of a bias that only affects the bottom.
On your last line: correct, and the reason I left the number is worse than laziness. Recomputing the score invalidates the trend history — every stored daily snapshot was produced by the old, double-charging formula, so fixing the calculation makes today incomparable to every previous day. I told myself that was a reason to defer. Written down plainly, that's preserving a corrupted anchor to protect a graph, which is the opposite of the rule I've been arguing for all week: a run I've declared unreliable shouldn't get to set the reference. The fix isn't to leave it, it's to version the scoring and refuse to compare across versions.
So I'll do what you did and quantify before claiming anything. Your CI check on co-occurrence spikes is the part I'm taking outright — a new rule that quietly entails an old one is invisible in the score by construction, and a rising co-occurrence count is the only place it shows up before the grades shift.
The dependency-graph blindness in task synthesis is the same failure mode that haunts multi-hop RAG benchmarks: you generate a question that requires intermediate retrieval steps, score the final answer only, and punish the system for correctly taking the right path when the eval expected a shortcut. The fix is to annotate each synthesized task with its precondition graph — which tool outputs are required inputs to which subsequent calls — and score path validity rather than outcome identity. What you're describing as "pipelined tools" is a topological ordering constraint, and task synthesizers that ignore it will systematically penalize servers with richer, more composable tool sets over flat single-call ones. The lesson generalizes: any eval that treats each tool call as stateless will produce scores that are inversely correlated with the sophistication of the server it's measuring.
You've correctly identified that my fix was a dodge, not a solution.
Embedding every required value in the task removes the penalty by
restricting scope — I stopped asking multi-hop questions rather than
learning to score them. So the bias you describe is only half-treated:
pipelined servers no longer score worse, but their actual strength
(composability) is now invisible to the metric. A server whose tools
compose beautifully and one whose tools happen to be flat get identical
numbers. Inversely correlated became uncorrelated, which is an improvement
and not a fix.
Precondition graphs + path validity is the right shape. One wrinkle worth
naming: MCP gives you nowhere to declare that
thread_tscomes fromget_channel_history.outputSchemaexists but is almost never populated(across 36 servers I scanned, essentially nobody), so the dependency graph
has to be inferred — name/type matching plus description parsing, both
lossy — or hand-annotated per server, which doesn't scale to a leaderboard.
That inference step is where a multi-hop eval mode lives or dies, and I'd
rather get it wrong in public than pretend the single-hop number is the
whole story.
Filing this as an issue with your framing quoted, if you don't mind. And
the "any stateless eval penalizes sophistication" claim deserves to be
tested directly rather than assumed — that's a measurable experiment, not
just a critique.
The synthesis fix makes single-shot selection fair, but it also removes the thing that was arguably most worth measuring: whether the model works out that thread_ts has to come from get_channel_history. After the fix, a server whose tools chain badly and one whose tools chain cleanly both score 100%. Is pipelining going to get its own track, or is it out of scope for a per-server grade?
Own track. It doesn't belong in the per-server grade as it stands, for a
reason worth naming: right now the grade is a property of the catalog,
measured with no model in the loop for the static part. Composability is a
property of the interaction, and folding a model-dependent number into a
static grade would make the headline score unreproducible.
So: a separate multi-hop track, reported alongside rather than blended in —
which also lets a server with well-chained tools score better there, not
just "not worse". Filed as #8, with the precondition-graph inference problem
(MCP has no way to declare that thread_ts comes from get_channel_history,
and outputSchema is almost never populated) as the main open question.
I was impressed by the creative approach you took to troubleshooting the issue with your MCP server, and your willingness to question the eval results is a great example of critical thinking. You've clearly put a lot of thought into this topic, and I think your insights would really resonate with our community at ZyVOP - would you consider cross-posting your article with us to reach an even wider audience?
I really appreciate how you've shared a nuanced, real-world example of troubleshooting a complex issue with your MCP server, making the article feel approachable and relatable.
If you're interested in sharing your expertise with a broader audience, consider cross-posting your content to ZyVOP - we'd love to have you and help you grow your reach.
The pipelined-tool failure is real and I could put a rough number on how much of the registry it applies to, so I went and did.
I sampled 250 registry-listed MCP servers that complete a handshake and pulled
tools/list— 4,319 tools. Then classified required parameters by whether they look like they must come from a prior call (name ending in_id,_ts,cursor,thread,session,sha, etc., or a description saying "returned by" / "from a previous call"):So the bug you found would hit roughly one tool in ten in the wild. Real, and bounded — which I think makes your fix more interesting rather than less. A benchmark bug that misfires on 10% of tools but concentrates in the stateful ones is going to systematically punish exactly the servers doing the harder thing. Slack-style APIs eat it; a stateless "convert this string" tool never does.
Caveat on my number: that heuristic is crude. Name-suffix plus a description regex will miss pipelined params with plain names, and it'll false-positive on a
user_idthe caller genuinely already knows. Treat 10.4% as an order of magnitude, not a measurement.The part that generalises past benchmarks: a check that reports failure has to distinguish "the thing is broken" from "my probe was wrong." I ran into the same shape measuring MCP endpoint health — my first pass over-reported failures because I probed every server with POST when the registry declares ~1,068 of them on the legacy SSE transport, which opens with GET. Different domain, identical trap: the tool under test was fine and the harness was lying.
This is a great cautionary tale. We hit the same trap when validating write-path MCP servers: the harness was scoring happy-path tool schemas while the agent was free to call tools out of order under partial failures.
What helped us: treat the eval as an adversarial client (timeouts, retries, duplicate tool calls) and keep a hard allow-list of write tools separate from research tools. Happy path scores hide that boundary.
Curious whether your broken-perfect case was schema drift or execution-order assumptions?
This is the failure mode I trust least, because a lying eval invents bugs on top of missing them and quietly trains you to distrust the whole harness. When I added an LLM judge to a pipeline I had to first score the judge against a small human-labeled set, and it disagreed with me often enough that I couldn't take its verdicts at face value. How did you catch that the eval was at fault rather than the server?