One question, 437,000 tokens: what real agents found in our MCP server

One question. 437,000 input tokens.

Not a hard question either. An agent connected to our MCP server, asked something a support engineer answers in a sentence, and worked its way there through twenty tool calls, each one dragging every earlier answer along behind it.

Nothing was broken while that happened. The server answered initialize correctly, spoke the 2025-03-26 revision, returned valid JSON-RPC to everything we threw at it. All of which turned out to be beside the point.

So we pointed real agents at production and watched. 18 scenarios, two vendors, a 5 dollar budget that we topped up once. This is the long version with the traces in it. There is a shorter one on our blog if you only want the conclusions.

What the server is

Briefly, because it shapes everything below.

FoxNose stores content as collections: schema-defined records with typed fields, some of them vector indexed. The MCP server is generated from that schema and served from the same URL prefix as the REST API. Fixed catalog of seven tools regardless of how many collections exist, five read and two optional write.

Two of those properties matter below. Collections are what an agent chooses between, so a badly described collection is effectively invisible. And the agent inherits exactly the rights of the API key it connects with, so there is no second allowed-tools list drifting out of sync with the first.

What the harness actually is

A scenario is a question in plain English, a set of tools, and a check.

The checks are where we made the most mistakes, so start there. They do not look at the answer text. Model output moves between runs, and a suite that asserts on wording is a suite you quietly stop trusting. They look at the trace: which tools ran, in what order, with what arguments, which errors came back, how many tokens the whole thing burned.

A check is a small predicate over the run:

any_of(
    no_tool_errors(),
    recovered_after("unknown_resource", then="search_records"),
)

That second branch exists because of a mistake I will get to at the end.

We ran everything twice, on Anthropic's hosted MCP connector and on OpenAI's hosted MCP in the Responses API. Same server, same questions, two clients.

Finding 1: the bill is round trips, not bytes

Back to those 437,000 tokens.

A hosted connector runs the tool loop on the vendor's side. Every iteration re-sends the whole conversation to the model. So a tool call does not cost what it returned. It costs what it returned, times the number of turns after it.

Input tokens per scenario, before and after we shipped truncated search results and changed the page size:

scenario	before	after
R4	437,000	78,000
N5	175,000	26,000
R5	128,000	17,000
R8	115,000	12,600

Same questions, same collections, same models. The only thing that moved is how much each call handed back and how many calls it took to get there.

Twenty calls against fourteen is not twenty over fourteen. Everyone tunes model choice. Almost nobody counts round trips.

What we shipped. Text fields in search results are capped at 1000 characters by default, and every field that got cut carries a marker with its path, its locale, and the original length. So the agent knows per field whether it is holding a fragment. An 11,917 character body arrives as 1000 and says so. get_record hands over all 11,917.

Page size went to a default of 5 with a maximum of 100, and both numbers live in the tool's JSON schema, so the model reads the range in the catalog before its first call.

The number was never the fix. Publishing the bounds was. We could not choose a good page size for everybody, so we stopped trying and told the caller the range instead.

Finding 2: your error strings are control flow

An agent hit an error, read the hint attached to it, did exactly what the hint said, failed the same way, read the same hint again. No crash. Nothing logged as wrong. It spent its entire iteration budget being obedient to a wrong instruction.

One line out of place caused it. We return structured errors, an error code plus a message plus a hint saying what to do next, and one code had been handed a different error's hint. Code right, status right, wrong sentence.

A person reads a wrong hint and shrugs. A model does not shrug. The hint is the next instruction.

Here is the same mechanism working, which is the part I actually want to show you. Real trace, from a scenario where the agent guessed:

discover_resources                                  ok
search_records  resource_id="documentation"         unknown_resource
                hint: call discover_resources and reuse exact resource_id
search_records  resource_id="kb_foxnose"            ok
get_record      MbJbdnzj9fJw                        ok

Four calls. It invented an identifier, got told exactly how to recover, recovered, answered. The whole exchange costs less than one wide search. Cheap recovery is something you build on purpose, and it only works when every hint points at its own failure.

Finding 3: a JSON-RPC notification is defined by the absent id

Small, pure protocol, which is why it lives here and not in the short version.

In JSON-RPC 2.0 a notification is a request with no id member. That is the entire definition. You must not reply to one, and you must reply to everything else.

Our server was deciding by method name. Anything starting with notifications/ got treated as a notification.

Those two rules agree nearly always, which is how it survived as long as it did. They part company when a client sends notifications/initialized with an id. That is a request. The spec says answer it. We sent silence.

No client we tested does this. It is the kind of thing that surfaces a year from now, in an integration you did not write, with a symptom that looks nothing like the cause. We key off the presence of id now, like the spec says.

Finding 4: the client decides your authentication

We had two auth schemes, both requiring you to write your own Authorization header. Never a problem, because every client we developed against was one where we wrote the header ourselves.

Hosted connectors do not work that way. One token field, always sent as Authorization: Bearer <token>. That is the whole interface. No configuration, no workaround.

So our server could not be used from the two clients most people reach for first. Not broken. Unusable. And invisible from the inside, which is the part that stuck with me.

What we shipped. Bearer tokens: opaque, 47 characters, no colon so they cannot be confused with a public:private pair, shown once at issue, bound to an existing API key and exactly as powerful as it.

Then we measured how long it actually takes to cut someone off:

deleting the key            took effect after 305 s
emptying the key's role     took effect after 284 s
revoking a bearer token     took effect on the next request

The first two are a permission snapshot cache expiring. Revoking the token skips it, because the token itself stops resolving. Worth knowing before the day you need it.

Finding 5: two vendors, two shapes

We wrote the OpenAI transport after the Anthropic one and assumed it was the same code with the nouns swapped. It would have been wrong in three places.

Tool call arguments arrive parsed in one and as a JSON string in the other, so you parse in one transport and must not in the other.

The OpenAI SDK types the error field on an mcp_call as an optional string. The wire sends a dict:

{"type": "mcp_tool_execution_error",
 "content": [{"type": "text", "text": "<json>"}]}

Trust the annotation and you will eventually call a string method on a dict, in a branch that only runs when something has already gone wrong. Great place for a second bug.

The third one will bite anyone building on the Anthropic connector. It runs at most 10 tool iterations per request. If the agent has not finished, you get stop_reason: "pause_turn" with the partial work. HTTP 200. No error field anywhere.

A client that checks only for end_turn records a success and shows a truncated answer. We caught it because a scenario answered half a question and the harness passed it.

Two things about the tests themselves

A check that forbids all errors forbids recovery. Our no_tool_errors check failed three scenarios that were completely fine. In each one the agent guessed, got a hint, corrected itself, answered. The trace higher up in this post is one of them.

We had spent real effort making guessing cheap, then written a check that punished guessing. That is where the any_of at the top comes from. If your surface is meant to be explored, your tests have to let it be explored.

A negative result is not a result. We checked that a key from one environment cannot read another. It could not, and I nearly wrote that down as proof of isolation.

Then we checked whether the key could read anything at all. "Access denied" looks identical whether the isolation held or the credential was already dead. This caught us three separate times: isolation, bearer auth, revocation. Each of those needed a positive control sitting beside it, a case you know should succeed. Without one, all you have proven is that broken things stay broken.

What it cost

Under ten dollars of API spend across about ten days. The harness is a few hundred lines. Reading traces took far longer than running them.

Final state is 18 out of 18 on both vendors, against production.

If you are shipping an MCP server: take five real questions, run them through a hosted connector, and read the traces rather than the answers. Count the calls. Look at what each one handed back. Then ask whether a model that cannot see your dashboard could have done better than that.