There's a controlled study going around that does something I wish more eval write-ups did: it holds the action contract fixed. Same executable capabilities on both sides — an MCP agent on one, a CLI agent on the other — and the behavior still split.
That shouldn't be surprising. The fact that it is tells you something about how we've been comparing interfaces.
The default comparison goes like this. Count the tools. Interface A exposes twelve tools; interface B exposes the same twelve operations as subcommands. Therefore the two are equivalent, and any difference in the results is noise or model flakiness. I've made that argument myself, in a design review, with a straight face.
It's a spec check. It tells you the surface area matches. It tells you exactly nothing about what the model does when it reaches for one.
The interface is a prompt
That's the part people skip. MCP isn't a wire protocol from the model's point of view — it's a block of text in the context window that happens to be generated from a schema. Tool names, descriptions, parameter docs, enum values: all of it sits in the prompt, every turn, whether the agent needs it or not.
A CLI is the opposite arrangement. Its documentation lives outside the context. The agent has to know a command exists, guess at its flags, or spend a turn discovering it — --help, a man page, an error message that says "unknown option". Sometimes that's cheap. Sometimes it's three turns of fumbling before anything happens.
So the two interfaces differ in at least four places that have nothing to do with capability:
Discovery. One gives you a menu in-context. The other makes you ask.
Error shape. A structured error object with fields is a different cognitive task from stderr text plus an exit code. With the second, the model has to read prose and decide whether to retry, escalate, or give up. That decision is where a lot of agent behavior lives, and it's entirely interface-dependent.
Composition. A shell lets you do in one call what takes four tool calls through MCP. Fewer calls is fewer places to fail, but a longer single string is more places to typo. Neither is free.
Irreversibility. A command that half-succeeds — writes two files, dies on the third — leaves you in a state no tool schema warned you about.
Hold capability constant and those four things still vary. Which is the study's point, and my point: capability parity is a precondition for a fair comparison, not the comparison itself.
What I actually built
I've been poking at this on and off for a few weeks with a harness that's smaller than I'd like. The design is boring on purpose.
One task spec, written once, in a format that knows nothing about transport. Two adapters behind it — an MCP server exposing the operations as tools, and a CLI binary exposing the same operations as subcommands, both calling the same underlying functions. Same model, same system prompt except for the interface section, same scripted environment so the world doesn't drift between runs.
Then I measure divergence, not just success:
- task success, pass/fail against a checked end state
- tool calls to completion
- retries after a failed call
- invalid-argument rate
- recovery rate after injected faults
- tokens and wall time
- action distance — how many steps between "agent states the intent" and "effect happens"
The fault injection matters more than I expected. Malformed JSON from the server. A call that times out mid-flight. Worst of all: exit code 0 with empty stdout. That one breaks agents in ways that look like model failure and are actually harness failure, and you only find it if you go looking.
My sample size is embarrassing and I'm not going to quote numbers from it as if they mean something. What I trust is the shape of the result: the two agents don't fail the same tasks in the same way. One over-calls. The other over-composes — long fragile command lines that work until a path has a space in it. Both eventually get there on the easy tasks. They diverge on the ones with a mid-run fault.
Maybe I'm wrong and a bigger sample collapses the difference. I'd like that, honestly. But I don't think it will, because the divergence isn't statistical, it's structural.
The eval consequence
Here's the thing that bugs me about published agent benchmarks. Most report one success rate per model. That number is a mixture over whatever interface the benchmark authors happened to wire up, and nobody reports the interface as a variable. So when a model scores 61% and someone else gets 48% on the same tasks, a chunk of that gap may be tooling, not capability.
You can't fix that by adding tasks. You fix it by treating the interface as a factor and running a behavioral parity suite: same task, same actions available, measured divergence across trajectories. If two interfaces genuinely produce the same behavior on your task distribution, great — pick whichever is cheaper to operate. If they don't, you now know what you're paying for, and it isn't the model.
The practical version of this for a team is small. Write your task specs transport-agnostic. Put the adapter boundary behind the tasks, not in front of them. Log every call with args and the raw response, including the ugly ones. Run the fault injection on day one, before you have a baseline you're attached to.
I'd rather find out in my own harness that the CLI agent collapses on a mid-run timeout than find out from a customer's bug report that says the agent "did something weird yesterday."
Pick the interface you can measure. That's the whole note.
Top comments (0)