Most MCP "readiness checklists" are really just a list of things that would be nice to have. They tell you to add logging, add auth, add tests — true, but not decision-grade. None of them answer the question a lead engineer actually asks before a release: given everything we know right now, do we ship this server to production agents or not?
This post is a rubric, not a wishlist. It's built to produce a number and a decision, the way a real go/no-go review works. It won't re-teach OAuth flows or walk through building a server from scratch — we've covered both of those in depth already. This is specifically about evaluation: how you score a server you (or someone else) already built, before it gets wired into an agent that real users depend on.
Why "it passed my manual tests" isn't an answer
Protocol-level tests (does tools/list return valid schemas, does the server respond to initialize) tell you the server speaks MCP correctly. They don't tell you what happens when an agent is calling it. Those are two different failure surfaces, and conflating them is the single most common mistake teams make when they say a server is "tested."
Server tests validate implementation behavior against known requirements, and MCP evals measure agent performance across realistic tasks. A server can pass every unit and conformance test and still fall apart the moment a non-deterministic agent has to choose between five similar-sounding tools, or retries a call that already partially succeeded. An MCP server can pass every protocol and handler test and still break down once an agent has to pick a tool, fill in valid arguments, and work through errors on the way to finishing a task. A scorer that reads only the final response misses silent failures and duplicate records left behind by a retry.
So the rubric below has two tiers, and you need both before you can call a server production-ready:
- Tier 1 — Protocol and handler correctness. Does the server do what its schema says, deterministically, every time?
- Tier 2 — Agent-facing behavior. When a real (or simulated) agent drives the server through a task, does it pick the right tool, construct valid arguments, and leave the system in a sane state afterward?
Tier 1: Protocol and handler checklist (score each 0/1)
Run these against the raw protocol, no LLM involved. They should be fast, deterministic, and part of CI — not something you eyeball once before a release.
| # | Check | Why it matters |
|---|---|---|
| 1 | Every tool's JSON schema is valid and matches the runtime handler's actual accepted inputs | Schema drift is invisible until an agent sends a payload the handler silently mishandles |
| 2 | Malformed input returns a structured error, not a stack trace or empty 200 | Agents parse error text to decide their next move; garbage in, garbage reasoning out |
| 3 | Every documented tool has at least one negative test (bad auth, missing field, out-of-range value) | Happy-path-only test suites are the norm, and it shows in production incident logs |
| 4 | Timeouts and partial failures are distinguishable from success | An agent that can't tell "still running" from "failed" will duplicate side effects |
| 5 | Token/session validation happens on every call, not just at connection time | Long-lived sessions with front-loaded auth checks are a common gap once bearer tokens get forwarded |
| 6 | p95 latency under realistic load is measured, not assumed | A tool that's 200ms in dev and 5s under load stalls the agent's reasoning loop and can trigger retries that compound the problem |
If you're scoring under 5/6 here, stop — don't move to Tier 2 yet. Agent-level evals on top of a shaky protocol layer just produce noisy, non-repeatable results.
Tier 2: Agent-facing eval checklist
This is the part almost nobody automates, because it requires actually running an agent against the server rather than curling endpoints.
| # | Check | Why it matters |
|---|---|---|
| 7 | Harness uses the same transport, auth path, and tool list as production | The eval harness acts as an MCP client, so it should use the same transport, authentication path, and exposed tools as the client used in production. A harness that skips OAuth or substitutes a shortened tool list is measuring a different configuration, and its scores say nothing about how reliably agents will use the production server. |
| 8 | Tool selection is scored separately from argument construction and outcome | A single pass-or-fail score shows a run failed but does not identify the cause. Effective agent evaluation separates tool use, arguments, execution, and outcome into distinct measurements. |
| 9 | Every task is run multiple trials, not once | Repeated trials: Multiple runs of the same task that reveal how consistently the agent completes it. A repeatable MCP eval ties each task-level pass rate to the agent behavior and system state behind it. |
| 10 | Adversarial and near-miss prompts are included, not just the intended use case | Test with adversarial prompts first. Don't just test the happy path. Try prompts that are similar to the intended use case but shouldn't activate the server. |
| 11 | System state after a run is checked, not just the final agent message | A trajectory can "look" successful while leaving duplicate writes or half-completed operations behind |
| 12 | Recovery behavior is tested, not just failure detection | Being observable isn't the same as being recoverable — check whether state can actually roll back or resume after a partial failure |
Score Tier 2 the same way: 0/1 per row, and treat anything under 5/6 as "not yet," not "close enough."
What "fine, it's not ready" actually looks like
A server that scores 4/6 and 3/6 isn't broken — it's early. The honest move is to say so publicly (or at least in your internal docs): note what's covered, what isn't, and what an integrator should assume is their responsibility until it's fixed. If the server has no concept of budgets, quotas, or side-effect governors, then it is asking the orchestrator to do all the defensive work. Sometimes that's acceptable. But then be honest: the server is not production-complete on its own. That kind of scoped honesty is more useful to the next engineer than a README that just says "production ready" with no evidence behind it.
Turning this into a repeatable process
The table above is something you can run once for a demo. The harder part — and the part that actually matters eight weeks from now when the server has changed twelve times — is making it repeatable: fixed task sets, versioned scorers, a way to diff "did this get better or worse after the last deploy" instead of re-litigating the whole checklist from memory every release.
That's exactly the gap our MCP Production Checklist + Eval Templates pack is built to close. It's the scored version of everything above, packaged as reusable YAML task definitions and a minimal harness scaffold you can point at any server — plus the Production Checklist itself as a standalone document your team can actually use in a release review instead of reconstructing it from a blog post every time.
If you're evaluating a server you didn't build — a community package, a vendor's offering, or a teammate's first attempt — running both tiers above before it touches real traffic is the difference between finding the gaps in a review meeting and finding them in an incident channel at 2am.
Written with AI assistance and reviewed for accuracy.
Top comments (1)
Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support