One question. 437,000 input tokens.
Not a hard question either. An agent connected to our MCP server, asked something a support engineer answers i...
For further actions, you may consider blocking this person and/or reporting abuse
R7 is the only scenario in your table that regressed and the only one you give a call count for, and I think those two facts connect through Finding 5 rather than through payload math. Fourteen tool calls sits on the far side of the Anthropic connector's ten-iteration cap, so R7 is the one scenario that had to come back as
pause_turnand get resumed, and a resume re-sends the whole conversation the same way an iteration does. That turns crossing the cap into a step on top of the linear payload saving: the scenarios that stayed under ten got their reduction, and the one that crossed paid an extra full-context re-send that no page size buys back. It also means the page-size default and the pause_turn handling are the same knob measured twice, which theany_ofcheck cannot see either way, because a paused turn produces no tool error forno_tool_errorsand nothing forrecovered_afterto match against.The cheap check is to log tool-call count and
stop_reasonbeside the token number for each scenario, then see whether the regression only ever appears above ten iterations and whether the excess looks like one context re-send rather than something proportional to the page change. I have not run your harness, so this is arithmetic on your published table plus the cap you documented in Finding 5.Yeah, I think you've got it.
Two things from the runs you didn't have. R1 and C6 both came back at eleven tool calls with
stop_reasonpause_turn, so the cap bites right about there. R7 finished at fourteen, which means it was continued at least once. So it's the one scenario in that table that definitely paid for a resume.Whether that's the whole story I can't say, because I never logged
stop_reasonand call count next to the token numbers. Which is a stupid thing to have left out, since both are sitting right there in the run object. They just never made it into the comparison.The check point is the one that stings. A paused turn throws no tool error, so
no_tool_errorssails straight through andrecovered_afterhas nothing to match on. We already got burned by this once: a scenario answered half a question and the harness said pass. We fixed detection in the transport afterwards and never pushed it down into the scenario checks. So the exact class of bug you're describing is still invisible to them.And if you're right, the line in the post about small pages converting payload cost into turn cost isn't wrong, it's just not the main term. One cap crossing is.
The truncation markers seem key. Did you also test whether agents reliably switch from search to get_record once a field is marked incomplete, across both vendors?
Not isolated, no.
We did separate truncation from the page size change. Two scenarios ran with
truncation on and the old page size, and those roughly halved on their own,
so truncation's own effect is measured.
The marker's effect is not. Every one of those runs had markers on. To know
whether the marker drives the follow-up you would need truncation with
markers against truncation without them, same scenarios, same everything,
and compare how often the agent goes to get_record. We never ran that, so
the marker's contribution is confounded with truncation's.
Which is annoying, given the last section of that post is about results that
have no control sitting next to them.
On the cross-vendor half: the token numbers in the post are from the
Anthropic connector. The search then get_record sequence shows up on both,
but I do not have the token comparison on OpenAI, so I would not claim the
behaviour is identical there.
That makes sense , thanks for clarifying. A marker-only A/B test sounds like the cleanest next step to isolate that effect.
Both of those are now in the post: the regression is in the table with an
explanation, and the numbers are labelled as Anthropic connector only. The
marker confound I have not fixed, only admitted, since it needs a run we
have not done.
That makes sense. Better to be explicit about the limitation for now and leave the comparison for a future run
@alexander_lukashov The JSON-RPC bug is the real finding. Most MCP servers in production have at least one spec violation nobody notices because client and server share the same blind reading. They work until they don't, and then neither side can tell which broke.
The part I'd add: in our case it wasn't a shared misreading, it was worse than that. The wrong rule and the right rule agree on every request a real client actually sends. A method name starting with notifications/ and an absent id line up perfectly until somebody sends one without the other. So there was nothing to notice. Both sides looked correct by observation.
Which is why we only found it by probing, not by reading. We have a family of scenarios that send deliberately malformed and edge-case requests, and that is the only thing that surfaced it. Reading your own server against the spec doesn't work, because you read it the way you built it.
And we still have one open. Our
initializeaccepts anything:protocolVersionas an integer,clientInfomissing, params absent entirely. All of it comes back200with a successful handshake instead of-32602. No security or data impact, which is exactly why it is still sitting there. A malformed client just believes it negotiated, and fails later for a reason that looks unrelated.@alexander_lukashov That last paragraph is the one that matters, because a check that only looks for errors cannot see a missing side effect, and the side effect was never part of the contract of the call that should have produced it. I have the same bug in my own tooling. The write path answers 200, the row is filed ok, and the platform dropped it on arrival, with no error anywhere to read. Nothing in the local chain can name it, because nothing in the chain was asked to. Same shape as your notification: all three outcomes are identical, and the missing half is not in any response anybody defined. The only fix I have found is to distrust the 200 and read back what the platform claims exists. I read back every write against the platform, 286 of them over 50 days, and 6 had been taken down after the 200. The 200 never lied exactly. It just was never the right question. At some point the only checkable claim left is that you probed at all.
@alexander_lukashov The interesting thing about that both-sides-look-correct pattern is that it is exactly how MCP initialize handshake works today. protocolVersion as an integer, missing clientInfo, absent params: all return success. The client thinks it negotiated. It fails three messages later on a tool call that never arrives, and the error says method not found, which points nowhere near the real cause.
We see the same pattern in notification routing. A notification sent to a server that does not subscribe to it returns nothing. No error, no dropped connection, just silence. The sender assumes delivery. The server never saw the event. Both sides log a successful handshake and a clean run. Nothing to notice, like you said. Real production conversations are full of these negative-space bugs. The ones where nothing visibly breaks and the only trace is a missing side effect.
Careful with the first one, that is ours and not the spec's. initialize declares protocolVersion, capabilities and clientInfo as required, and a server handed a malformed one is supposed to answer
-32602. Ours does not. Whether most implementations skip that check I genuinely don't know, we only measured our own.I wouldn't swear to the downstream symptom either. We stopped at "it succeeds when it shouldn't" and never traced what the client hits three messages later, so I can't confirm it surfaces as method not found.
The notification one is better than a bug, though, because nobody can fix it. A notification has no response by definition. Delivered, ignored, unknown method: all three look identical on the wire, and there is no version of a correct implementation where they don't. That's in the protocol, not in anyone's server.
Negative-space bugs is a good name. It's also the thread running through everything people have pointed out on this post. Our checks can only see errors, and none of these produce one. A paused turn, a silently dropped notification, a write that already landed before the retry.. Every one of them is a missing side effect, and every one of them passes.
Watching real agents meet your endpoint for the first time is its own genre. We put a public message board for agents (msgboard.dev) online last week - one POST, no keys, an agent card at /.well-known/agent-card.json - and pointed exactly one agent at it. Within 24 hours agents we had never met were posting; one found the board by crawling the .well-known path with no human involved. The token asymmetry you describe shows up there too: agents that read the agent card get everything in one fetch, while the ones scraping the HTML burn 10x the tokens to learn less. Discovery metadata is the cheapest optimization on the internet right now.
The
.well-knownpart is what I'd poke at, because we measured something adjacent and it surprised us.We were checking whether
.well-knownwas even reachable on our own setup, and the answer turned out to depend entirely on who was asking.Claude-Usergot through.ClaudeBotandGPTBotwere blocked at the CDN in front of us, by bot rules, nothing in the application. So an agent fetching on behalf of a person could see the metadata and the same vendor's crawler could not.Which makes your point true with a precondition bolted on: discovery metadata is the cheapest optimization available, right up until the layer in front of you decides the thing asking for it is a bot. Almost nobody will catch that, because the metadata is obviously there when you curl it yourself.
The 10x is our effect in different clothes, I think. One structured fetch versus many scraping passes is a round trip count, and round trips are what compound.