Originally published at tengli.dev
When I added an LLM-powered eval to mcpgrade, the first real run produced a result that looked like a scoop: co...
For further actions, you may consider blocking this person and/or reporting abuse
"The benchmark was grading correct multi-step reasoning as failure" is the part worth sitting with, because the eval wasn't just wrong — it was wrong in the direction that punishes the right behavior. A model that declined for lack of a thread_ts was doing the safest possible thing, and your grader scored it below a model that would have hallucinated a timestamp. An eval that rewards confident fabrication over honest refusal is optimizing for exactly the failure you least want in production.
I hit the same shape building the thing that's supposed to catch this. I seed my own security scanner with known-bad files to prove it can see, and my test originally asserted "a finding fired" rather than "the specific finding I planted fired." So a fixture that tripped the wrong rule counted as a pass — the grader was checking for activity, not correctness, same as yours checking for a final answer instead of the right tool sequence. The tell was identical too: one category of input scored suspiciously well (your single-step server at 93%, my fixtures that happened to trip any rule), and that outlier was the eval's bias showing, not the system's competence.
What fixed it for me was asserting the trajectory, not just the terminal state — the finding has to match the id I planted, and a red has to name which specific thing failed. Your multi-step version is harder because the correct path branches, but the principle survives: the grader has to model "get_channel_history then post_message" as a valid trajectory, not score the endpoint. Otherwise the benchmark is measuring whether the model shares its author's assumption about how many steps the task takes, which is a fact about the eval, not the server.
"Checking for activity, not correctness" — that's the cleanest statement of
this bug I've read, and your fixture case is the same shape one layer down:
you asserted that a finding fired, I asserted that an answer came back.
The tell being identical is the part I keep thinking about. One category
scoring suspiciously well is a signature, not good news. My single-step
server at 93% read as "this one's fine" when it was actually "this one is
the only case my grader can handle." Worth generalizing: in any eval, the
subgroup that scores best is the one whose shape your grader assumes.
Trajectory-not-terminal-state is where this has to go. The branching is the
hard part — several correct paths exist, and a grader that enumerates them
is just a slower version of the same assumption — so the interesting version
scores each step against what the model knew at that point, rather than
matching against a blessed sequence. That's #8 on the repo if you want to
argue the design; you clearly have scar tissue I'd like to borrow.
Your subgroup generalization holds in my data too, and it showed up as the opposite-looking symptom. One of my seeded vulnerabilities tripped five separate rules while most tripped exactly one. I read that as "this category is well covered." It actually meant my rules overlapped on that pattern — the same detection wearing five names, which is a noise problem, not a coverage win. Highest score, worst signal, exactly as you put it: the outlier was telling me about my grader, not my scanner.
On step-wise scoring, the design question I'd raise is where "what the model knew at that point" comes from. If it's reconstructed from the model's own trace, the criterion is being derived from the artifact under test, and that collapses in a specific way: a model that skipped a prerequisite now looks correct at every step, because its knowledge state at step N is inferred from what it actually did at step N-1. Wrong trajectories become self-justifying. This is the same failure as re-pinning a fixture's expectations to the tool's current output — the assertion silently changes from "this proves X" to "this is what the thing does," and it reads as a passing test forever after.
What kept it separable for me was declaring intent in the fixture and forbidding derivation from output. Each planted vulnerability carries a tag naming the rule it must trip; the tag is authored with the fixture, never re-derived from a scan, and the surrounding cluster of incidentally-tripped rules is tracked separately as a change detector. The two claims can't be merged — and because a comment in the code wasn't enough to hold that, there's now a test that performs the forbidden merge on purpose and asserts the intent check still goes red. For your case that translates to: the task fixture declares each step's preconditions when the task is written — thread_ts is unavailable until get_channel_history has returned — and the grader scores only against those declarations. Multiple correct paths are fine; what can't be allowed is the model's trace teaching the grader what was available. Happy to argue it on #8 — that's the assumption I'd stress-test first.
The circularity point lands, and it's the failure mode I would most likely
have shipped. If step N's "available knowledge" is reconstructed from the
trace, then a model that skipped get_channel_history has a knowledge state
at step N inferred from what it did at N-1 — so every step it took looks
legal and the wrong trajectory certifies itself. The grader stops testing
the model and starts describing it. Same shape as re-pinning a fixture to
current output: the assertion quietly changes from "this proves X" to "this
is what the thing does," and it's green forever.
So: preconditions declared when the task is authored, never derived from a
response. thread_ts is unavailable until get_channel_history has returned,
and that's a property of the fixture, not of the run. Multiple correct paths
stay fine; what's forbidden is the trace teaching the grader what was
available. And your enforcement trick is the part I'd have skipped — a test
that performs the forbidden merge on purpose and asserts the intent check
still goes red. A comment can't hold an invariant that convenient.
Your overlap finding sent me to check my own scorer, and it's there too.
On firecrawl: of 135 flagged parameters, 46 trip both D004 (no description)
and S008 (complex param, no example). That's one defect — "this parameter is
undocumented" — priced twice, and since the score is penalty-over-capacity,
34% of that catalog's parameter findings are double-charged. Highest finding
count, worst signal: I read 199 findings as "thoroughly diagnosed"
The double-charge is worse than a scoring error because of where penalty-over-capacity sends you next. A catalog whose defects happen to trip overlapping rules doesn't just score lower — it outranks catalogs that are genuinely worse, so remediation effort flows toward the shape your ruleset over-counts rather than toward the biggest real problem. The measurement error becomes a prioritization error, and that one is invisible in the score because the score is exactly what's producing it.
Looking at your pair, D004 and S008 don't read as independent to me — they look like an implication. If a parameter has no description at all, it necessarily has no example; "undocumented" strictly contains "no example for a complex param." So those 46 aren't two findings that coincide, they're one finding observed twice at different granularities, and the general form is that a ruleset without declared implications between rules will price correlated defects as independent ones. Worth asking of every pair that co-occurs above chance: is this coincidence, or does one entail the other? The entailed ones should collapse; only genuinely orthogonal co-occurrence deserves two charges.
And your finding sent me back the same way, with a less flattering result than I gave you. I identified the overlap in my fixtures — one seeded vulnerability tripping five rules — and I pinned the cluster so the test would notice if it changed. What I did not do is dedupe it in the scoring. My scanner still computes per-dimension scores by subtracting per-finding penalties, so a program with one hardcoded secret takes five deductions for one mistake, and the dimension weights then propagate that inflation into the composite. So I found the same defect you did, wrote a test around it, and left the number it corrupts uncorrected — which is a nice demonstration that noticing a bug and fixing what it affects are separate pieces of work. Yours is the more honest write-up: you quantified the damage. I only labeled the cause.
Ran the audit you implied. It's an entailment, and the damage is worse than
the count suggested — but in a more specific direction than "prioritization
error."
D004 fires on "no description." S008 fires on "complex param, no examples,
description lacks e.g." A parameter with no description trivially has no
"e.g." — so for any complex param without an examples field, D004 entails
S008 outright. Not correlated: implied. The only escape is a param that
populates examples but no prose.
Across 12 servers, D004+S008 co-occur on 75 subjects. Next-highest pair: 7.
Collapsing the entailment:
firecrawl 58 F → 63 D (46 dropped) ← grade flip
mongodb 66 D → 67 D (14)
github 67 D → 68 D (5)
memory 78 C → 79 C (4)
slack, maps, shrimp, elasticsearch, airbnb, context7, todoist: unchanged
The pattern is that it's regressive, not random. No rank inversions in
this sample — but no A-grade server is affected at all, because a catalog
that documents its parameters never trips D004 in the first place, so it can
never be double-charged. The error concentrates entirely on the servers
already at the bottom, and it crossed a letter boundary on the one server I
used as the poster child in the launch post. F and D produce different
reactions from a maintainer.
So the general form you named holds and gets sharper: a ruleset without
declared implications doesn't misprice uniformly, it over-weights whichever
defect class happens to have the most redundant coverage — and that
over-weighting lands hardest on whoever has that defect most. Mine pointed at
a real problem, which is luck, not design.
Audit + data + repro script: docs/rule-overlap-audit.md. Making it a CI check
too, so a new rule that quietly entails an old one shows up as a co-occurrence
spike instead of a silent penalty increase.
And re: your scanner — you found it, tested the invariant, and left the number
uncorrected.
The dependency-graph blindness in task synthesis is the same failure mode that haunts multi-hop RAG benchmarks: you generate a question that requires intermediate retrieval steps, score the final answer only, and punish the system for correctly taking the right path when the eval expected a shortcut. The fix is to annotate each synthesized task with its precondition graph — which tool outputs are required inputs to which subsequent calls — and score path validity rather than outcome identity. What you're describing as "pipelined tools" is a topological ordering constraint, and task synthesizers that ignore it will systematically penalize servers with richer, more composable tool sets over flat single-call ones. The lesson generalizes: any eval that treats each tool call as stateless will produce scores that are inversely correlated with the sophistication of the server it's measuring.
You've correctly identified that my fix was a dodge, not a solution.
Embedding every required value in the task removes the penalty by
restricting scope — I stopped asking multi-hop questions rather than
learning to score them. So the bias you describe is only half-treated:
pipelined servers no longer score worse, but their actual strength
(composability) is now invisible to the metric. A server whose tools
compose beautifully and one whose tools happen to be flat get identical
numbers. Inversely correlated became uncorrelated, which is an improvement
and not a fix.
Precondition graphs + path validity is the right shape. One wrinkle worth
naming: MCP gives you nowhere to declare that
thread_tscomes fromget_channel_history.outputSchemaexists but is almost never populated(across 36 servers I scanned, essentially nobody), so the dependency graph
has to be inferred — name/type matching plus description parsing, both
lossy — or hand-annotated per server, which doesn't scale to a leaderboard.
That inference step is where a multi-hop eval mode lives or dies, and I'd
rather get it wrong in public than pretend the single-hop number is the
whole story.
Filing this as an issue with your framing quoted, if you don't mind. And
the "any stateless eval penalizes sophistication" claim deserves to be
tested directly rather than assumed — that's a measurable experiment, not
just a critique.
The synthesis fix makes single-shot selection fair, but it also removes the thing that was arguably most worth measuring: whether the model works out that thread_ts has to come from get_channel_history. After the fix, a server whose tools chain badly and one whose tools chain cleanly both score 100%. Is pipelining going to get its own track, or is it out of scope for a per-server grade?
Own track. It doesn't belong in the per-server grade as it stands, for a
reason worth naming: right now the grade is a property of the catalog,
measured with no model in the loop for the static part. Composability is a
property of the interaction, and folding a model-dependent number into a
static grade would make the headline score unreproducible.
So: a separate multi-hop track, reported alongside rather than blended in —
which also lets a server with well-chained tools score better there, not
just "not worse". Filed as #8, with the precondition-graph inference problem
(MCP has no way to declare that thread_ts comes from get_channel_history,
and outputSchema is almost never populated) as the main open question.
I was impressed by the creative approach you took to troubleshooting the issue with your MCP server, and your willingness to question the eval results is a great example of critical thinking. You've clearly put a lot of thought into this topic, and I think your insights would really resonate with our community at ZyVOP - would you consider cross-posting your article with us to reach an even wider audience?
I really appreciate how you've shared a nuanced, real-world example of troubleshooting a complex issue with your MCP server, making the article feel approachable and relatable.
If you're interested in sharing your expertise with a broader audience, consider cross-posting your content to ZyVOP - we'd love to have you and help you grow your reach.
The pipelined-tool failure is real and I could put a rough number on how much of the registry it applies to, so I went and did.
I sampled 250 registry-listed MCP servers that complete a handshake and pulled
tools/list— 4,319 tools. Then classified required parameters by whether they look like they must come from a prior call (name ending in_id,_ts,cursor,thread,session,sha, etc., or a description saying "returned by" / "from a previous call"):So the bug you found would hit roughly one tool in ten in the wild. Real, and bounded — which I think makes your fix more interesting rather than less. A benchmark bug that misfires on 10% of tools but concentrates in the stateful ones is going to systematically punish exactly the servers doing the harder thing. Slack-style APIs eat it; a stateless "convert this string" tool never does.
Caveat on my number: that heuristic is crude. Name-suffix plus a description regex will miss pipelined params with plain names, and it'll false-positive on a
user_idthe caller genuinely already knows. Treat 10.4% as an order of magnitude, not a measurement.The part that generalises past benchmarks: a check that reports failure has to distinguish "the thing is broken" from "my probe was wrong." I ran into the same shape measuring MCP endpoint health — my first pass over-reported failures because I probed every server with POST when the registry declares ~1,068 of them on the legacy SSE transport, which opens with GET. Different domain, identical trap: the tool under test was fine and the harness was lying.
This is a great cautionary tale. We hit the same trap when validating write-path MCP servers: the harness was scoring happy-path tool schemas while the agent was free to call tools out of order under partial failures.
What helped us: treat the eval as an adversarial client (timeouts, retries, duplicate tool calls) and keep a hard allow-list of write tools separate from research tools. Happy path scores hide that boundary.
Curious whether your broken-perfect case was schema drift or execution-order assumptions?
This is the failure mode I trust least, because a lying eval invents bugs on top of missing them and quietly trains you to distrust the whole harness. When I added an LLM judge to a pipeline I had to first score the judge against a small human-labeled set, and it disagreed with me often enough that I couldn't take its verdicts at face value. How did you catch that the eval was at fault rather than the server?