Every MCP server I have written passed its tests before it was any good.
That is not a joke about test quality. The tests were fine. list_tools returned the
right shape, every handler round-tripped its arguments, the JSON-RPC framing was
correct, errors serialised. Green across the board. And then the model on the other
end would call search(query="...", max_results=5) when the parameter was named
limit, or call the tool three times in a row with identical arguments because the
first response didn't obviously say it had succeeded, or never call it at all.
None of that is a bug in the server. All of it is a defect in the product. This is the
thing I keep having to re-learn: an MCP server's real interface is not the wire
protocol. It is the description text, and nothing in your test suite touches it.
The untested surface
Look at what a model actually receives from your server. Not the code — the payload:
- the tool name
- the one-line description
- the JSON Schema, including each property's
description - whatever your last response said
That's it. There is no README in the context window. There is no example usage unless
you put it in the description. The model is integrating against your API having read
exactly the fields above, once, in a list alongside forty other tools.
Now look at what your tests assert. Almost certainly: given arguments X, the handler
returns Y. Which is the one part of the system that was never in doubt.
The gap between those two lists is where all my time has gone.
Three defects that pass every test
The name that lies slightly. I had a tool called get_status. It returned the
status of a job, not of the server. Perfectly documented in the schema description.
The model called it whenever it wanted to know if anything was healthy, because the
name is what gets pattern-matched under load and the description is what gets skimmed.
Renaming it to get_job_status fixed a class of misuse that no amount of description
prose had fixed. Names are load-bearing. Treat renaming as a real fix, not cosmetics.
The optional parameter that isn't. A schema said path was optional, defaulting to
the workspace root. Sensible. In practice the model omitted it constantly and got back
a 400-item listing that ate the context window, and then apologised and tried again.
The schema was honest and the default was wrong. "Optional" in JSON Schema means the
call is valid without it. It does not mean the call is useful without it, and the
model has no way to tell those apart from the schema alone. Either make it required or
make the default genuinely cheap.
The success that reads like a failure. A write tool returned {"ok": true}. The
model would frequently re-issue the write. Why wouldn't it? The response contained no
evidence that anything had happened — no path, no byte count, nothing to quote back to
the user. Returning Wrote 1,204 bytes to /home/x/notes.md stopped the retries
immediately. Responses aren't just control flow; they're what the model reasons over on
the next turn. A response that can't be reasoned over gets retried.
Notice that all three are content defects living in string fields. Type-checked,
schema-valid, and wrong.
What I do instead now
I have stopped thinking of this as testing and started thinking of it as evaluation,
which is an uncomfortable admission because evaluation is slower, fuzzier and harder to
put in CI. But the alternative is shipping a server that is provably correct and
practically unusable.
Read the tool list as the model sees it. Dump exactly what goes over the wire in
tools/list and read it cold, as a flat list, with no knowledge of your codebase.
Every time I do this I find two tools whose descriptions only make sense if you already
know which one you want — which is the one thing the reader doesn't know.
Keep a small set of intent transcripts. Not "call tool X with args Y" but "given
this user request, did the model reach the right tool on the first call?" Ten of these,
run by hand before a release, catch more than a hundred handler assertions. They are
noisy and non-deterministic. They are also the only tests that exercise the actual
interface.
Assert on descriptions in CI. This part is mechanisable, and it's the highest
value-per-line test I write: every tool has a non-empty description; every property has
a non-empty description; no description is under N characters; names match a convention.
It catches nothing subtle and it catches the boring regression where someone adds a
parameter and doesn't document it, which is the most common way a good server decays.
Treat repeated calls as a bug report. When I see a model call the same tool twice
with the same arguments, I no longer read it as the model being careless. It is nearly
always my response failing to convey that the work was done. The model is the smoke
detector; the fire is in my output formatting.
The uncomfortable part
There's an instinct — I have it strongly — to treat the model's misuse of a correct API
as the model's problem. It's a defensible position and it is also a losing one, because
you can't file a bug against the caller. The caller is a probabilistic system that read
your schema once. If it consistently misreads you, the schema is ambiguous, whatever
the type checker thinks.
The mental shift that helped: your MCP server is not a library. It's closer to a CLI
designed for someone who will never read --help twice and cannot ask you a question.
Everything that person needs must be legible at the moment of the call. That's a
documentation problem wearing a protocol's clothes, and it will not show up green or
red in your test runner.
I write about agent tooling and the boring engineering underneath it. Some of this
material is worked through in more depth in my book,
Building Production MCP Servers.
Top comments (0)