I built an evaluation suite for an MCP server — the kind that hands a product's data to an AI agent as a set of tools. This is the shape it ended up with and the reasoning behind each piece, with the product specifics stripped out. Not a framework you can install; a structure you can copy.
The short version: MCP eval is messier than agent eval, and it is messier for a reason you can design around once you name it.
The simplest way to think about the suite is:
- deterministic checks test whether the server works;
- agent runs test whether a model can navigate it;
- an LLM judge reviews whether the server's design helped or hindered that navigation.
Why MCP eval isn't agent eval
Agent eval is well-trodden. Fixed task, known answer, score the output, run it enough times to average out the noise.
MCP breaks that in a specific way: the server is deterministic and the caller isn't yours.
Your MCP server is ordinary software: same arguments in, same bytes out. Most of its behavior can be tested deterministically, though protocol and deployment checks still need their own layer. All the nondeterminism lives in the model on the other end — a model you did not train, cannot pin, and which the user replaces the day they switch clients.
So when an agent gives a bad answer through your tools, at least four different things could be true:
- Your tool returned the wrong data. Your bug.
- The agent never found the right tool. Your discovery surface.
- The agent found it and called it wrong. Maybe your names, maybe their model.
- The agent had good data and reasoned badly. Not your product at all.
One pass/fail score collapses all four into a number you cannot act on. Worse, the number moves when the vendor ships a new model and you change nothing.
Two consequences drove everything else I built:
- Separate the layers by what they need to run, not by what they're called.
- Grade properties that name a repair site. "The agent scored 7/10" is a vanity metric. "The agent couldn't tell from the name what this tool returns" is a rename.
The layer stack
Four layers. They fail for different reasons and cost different amounts, which is exactly why they stay separate.
| Layer | Asks | Needs | Can it fail a build? |
|---|---|---|---|
| L0 unit | does the code do what it says | nothing | yes |
| L1 contract | does a deployed server return the right thing | a server + a token | yes |
| L2 navigation | does an agent reach for the right tool | a server, an agent CLI, money | yes, grudgingly |
| L3 answer review | did our design let the agent reach a defensible answer | a stored run + a judge | no |
L0 is normal testing. Nothing interesting, and that's the point — most of what people try to catch with expensive agent runs is a unit test.
L1 is the one teams skip and shouldn't. It speaks the protocol to a deployed server with no agent involved: does tools/list match in both directions, do IDs come from discovery rather than fixtures, do windowed tools echo the window they used, and do errors stay errors? It's free, deterministic, and catches contract drift before an agent run.
L2 is where an actual agent gets an English question and you grade the path it took. This is the expensive, flaky layer.
L3 reads a run that already happened. That offline property is the best decision in the whole suite: changing a rubric re-scores all the history you already have instead of costing another sweep, and judging a stored run costs cents against the sweep's dollars. Iterate on the grader for the price of coffee.
The trajectory has to come from the server
Here's the mistake I nearly made: grade L2 by parsing what the agent CLI says it did.
Don't. The agent is the thing under test. Its summary of its own behaviour is a generated artifact like any other — it drops calls it made and regretted, compresses retries, and occasionally describes a call it never issued.
Your server already knows. Log one row per tool call and read the trajectory back out of that.
agent ──calls──▶ MCP server ──writes──▶ tool_call log
│ │
└──✗ "here's what I did" ──▶ grader ◀──────┘
(self-report) (ground truth)
This is just ordinary observability — one row per call with the tool name, an allowlisted subset of the arguments, the timestamp and a session id. You want it anyway, to know who uses your server. It turns out to be the substrate the whole eval sits on.
Two things I'd repeat:
- Fire-and-forget the write. A logging hiccup must never fail a customer's tool call, and the log must add no latency to the happy path. You lose rows on a crash; that's correct for analytics and fine for eval.
- Allowlist the arguments. Record that a search happened and which fields were filtered, not the free-text query. You're going to look at these rows a lot, and they are full of other people's data.
With the trajectory coming from the server, the graders get interesting: tool choice, call order, argument agreement, how many discovery calls happened before the first real one, and which tools were never touched at all. None of that needs an LLM.
Scenarios and probes do different jobs
I ended up with two banks, and I'd resisted the second one for a while.
Scenarios are deep. A realistic question a user would actually ask — "which audience has the worst failure rate this month, and why" — with assertions about the path: these tools, roughly this order, with these arguments. A scenario proves an agent can complete a task.
Probes are shallow and wide. One question per tool, aimed at getting that tool called at all. A probe proves a tool is reachable.
Why both? Because scenarios systematically miss a whole failure class. Scenarios are written from tasks you already know matter, so they exercise the tools you already think about. A tool nobody wrote a scenario for can be broken for months in perfect silence. My probes found exactly that: a breakdown tool that advertised a long list of dimensions and returned an empty result for most of them, because only a handful had ever been wired up. Every scenario passed the whole time — none of them happened to ask for the unwired ones.
Probes also produce the cheapest useful metric in the suite: tool coverage. What fraction of your tools did anything ever call? A tool at zero is either undiscoverable, badly named, or should be deleted.
Five rubrics, each naming where the fix lands
L3's judge is an LLM reading the agent's final answer alongside its real trajectory. The single most important rule:
The judge is not asked to independently verify the answer's numbers.
The payload is deterministic, so numerical correctness belongs to L0 and L1. The judge instead asks whether the server's surface gave the agent enough information to produce a defensible answer: could it find the right tool, understand the names, interpret the result, and recognize missing data? Grading the model's reasoning itself measures someone else's product and tells you little you can fix.
So the judge grades five properties of your surface, and each one is named after the place a failure gets repaired:
| Rubric | A failure here is fixed in | |
|---|---|---|
| D1 | Discoverability — could the agent find what exists, and learn what doesn't | discovery |
| D2 | Nameability — did your names predict what they return | a rename |
| D3 | Result self-description — did the response say what it actually was | the payload |
| D4 | Guidance sufficiency — did the rules you wrote reach the answer | your own prose |
| D5 | Dead-end honesty — when a call returned nothing, does the answer say so | the payload |
The last two earn their place more than I expected.
D5 exists because "no data" and "no problems" look identical in most payloads. A tool that returns 0% for an unmeasured metric is lying, and the agent will faithfully repeat the lie. Return null plus a sample size and the distinction survives the trip.
D2 caught the best bug in the set. A user asks for a breakdown "by audience"; the dimension is called cohort in the schema. Both words were live in the product, on different surfaces, for the same field. No test could have caught it because nothing was broken — the names just disagreed, and only a reader trying to get from the question to the tool would notice.
Three agents, one server
This is the part I'd push hardest, because it is relatively cheap compared with misdiagnosing your own server.
When you can, run the same scenarios through two or three different agent clients or models. One flag: --client a|b|c. Same server, same questions, same graders, trajectory from the same log.
You're not benchmarking the agents. You're separating your bugs from their behaviour, and you cannot do that with one client no matter how many times you run it.
Two findings that a single client could not have produced:
A tool went uncalled in every sweep. With one client, this looked damning — a tool so badly named or described that nothing ever reached for it. I was ready to redesign it. Then the second client called every tool in the set, including that one. It wasn't a design defect; it was one model's habit. Had I only run the first client, I'd have spent time fixing something that worked.
A client omitted a required argument entirely, calling the breakdown tool with no dimension at all. The other two passed the identical question. That's a client not following a schema — worth knowing, worth a workaround, but not evidence that the surface is confusing.
Same observation, opposite conclusions. The only thing that distinguishes them is a second and third subject.
A caveat I'd write in your own report too: swapping the CLI usually swaps the model along with it, so "client A used fewer calls" really means "client A on that model used fewer calls." Don't over-read the differences. The value is in the disagreements, not the ranking.
Two doors, one dataset
Most products that expose an MCP server also have an assistant built into the app. Same data, same questions, different door.
The instinct is to build two eval setups. Resist it.
┌─▶ MCP door ─┐
one scenario ────┤ ├──▶ same tool_call log ──▶ same rubrics
└─▶ in-app door ┘
The trick is writing scenarios as questions a user asks, not as sequences of protocol calls. "Which audience has the worst failure rate this month" is answerable through either door. Once your scenarios are phrased that way, the door becomes an adapter and everything downstream — trajectory, graders, rubrics, reports — is shared.
You get two things from that:
Attribution. A rubric that fails through both doors is your design. A rubric that fails through one is that door's implementation. Without the comparison you can't tell, and you'll argue about it in review instead of resolving it.
Enforced consistency. This is the part I underrated. When both surfaces are graded by one dataset, drift between them becomes a test failure rather than a discovery. Your in-app assistant and your MCP server should agree about what a dimension is called, which metrics exist and what an empty result means — they're the same product. Two separate eval suites let them diverge quietly and each stay green. One shared suite makes staying in sync a requirement, which is how it should have been anyway.
I've built the shared layer and one door. The second adapter is the payoff I haven't collected yet — but the design is what forced the scenarios to be written in user language, which improved them regardless.
Keeping a judge you can trust
An LLM judge can be confidently wrong. These guards keep a bad judgment from looking like a reliable test result:
| Guard | What it prevents |
|---|---|
| Calibration fixtures | The judge drifting after a prompt, model, rubric, or vocabulary change. Hand-label clear passes, clear fails, and a few ambiguous cases; run them before trusting a sweep. |
| Require quotes for failures | The judge inventing evidence. A fail must quote text that appears verbatim in the answer; otherwise the harness downgrades it to unclear. This protects against fabricated failures, not fabricated passes. |
| Missing verdicts are not passes | A broken or incomplete judge response producing a green report. A skipped rubric is unclear; only an explicit pass passes. |
| Report-only | LLM randomness blocking the build. The judge reports observations for humans; deterministic checks remain responsible for build failures. |
One more rule: keep the judge blind to deterministic verdicts. If it knows that L2 passed, it may simply agree. Because the judge reviews stored runs offline, it can inspect the answer and trajectory without seeing the earlier verdict.
If you remember only three things
- The server is deterministic; the caller isn't yours. Separate layers by what they need to run, so the free deterministic checks aren't held hostage by the expensive flaky one.
- Read the trajectory from your own logs, never from the agent's self-report. The subject under test doesn't get to file the report.
- Grade properties that name a repair site. Discoverability, nameability, self-description, guidance, dead-end honesty — each failure points at one place to fix.
Closing
I went in expecting to measure answer quality and came out measuring something more useful: whether a literal-minded reader can get from a plain English question to the right tool, with the right arguments, and know what it didn't get. That's a property of your design, and every failure comes with an address.
Start smaller than I did. A tool-call log, a handful of probes — one per tool — and the coverage number. If any tool sits at zero, you've learned something before writing a single rubric.
Top comments (0)