Same Haiku 5.5 Scores 82% or 65% on MCP Servers: The Harness Decides

On 1 November 2026 Sineframe published numbers from their M3 evaluation suite: the same Claude Haiku 5.5 scored 82.0% across 21 MCP servers when driven by OpenCode 1.19.5 and only 65.4% when driven by Claude Code 2.1.312. The gap was not model reasoning. It was the harness.

They split the suite into 11 servers with no native-tool overlap (Postgres, MongoDB, Redis, Kafka, Linear, Jenkins, Grafana, Kubernetes, Gitea, two browser engines) and 10 servers that collided with tools the agent already ships. On the non-overlapping set the three agents sat within 4 points of each other (OpenCode 96.2%, Claude Code 93.5%, Codex/GPT-6 Luna 92.1%). On the overlapping set OpenCode dropped to 66.3% while Claude Code collapsed to 34.5% and Codex to 17.2%. Across the full 21 the totals were 82.0% / 65.4% / 56.5%.

The model was held fixed. The only variables that moved were which tools the client already advertised, how those tools were named and described, and which ones the agent preferred to call first. When an MCP server and a built-in tool both claim “query the database” or “open a browser”, the model picks the one whose description matches the prompt most closely. The one it picks first is the one that already lives in its system prompt, not the one you just attached.

Why the harness eats the model

MCP tools arrive as a list of JSON schemas. The host injects them into the model’s context, usually as a system or tool-use message. If the host also injects its own tools with overlapping names or descriptions, the model sees two near-identical function calls and has to choose. Most agents do not run a separate ranking step; they simply let the next token decide. The next token is heavily biased toward whatever was already in the prompt when the session started.

OpenCode’s advantage in the benchmark appears to come from a lighter built-in tool surface and a clearer preference order. Claude Code’s larger set of always-on tools (file system, shell, browser, search, etc.) creates more collision surface. Codex’s even larger surface creates still more. The evaluation does not claim one agent is “smarter”; it shows that the same weights produce different outcomes once the tool list changes.

This is the same pattern that appears in every other agent benchmark that holds the model fixed and varies the scaffolding: tool descriptions, retrieval order, context compaction, and the exact set of functions the model is allowed to call. The 17-point swing is just the latest public measurement.

Practical consequences for anyone shipping an agent

If you are building a coding agent, a desktop automation agent, or a real-time voice agent, the MCP servers you attach are only half the story. The other half is the tools you already expose.

A few concrete patterns that reduce the collision:

• Prefer tool names that cannot collide. mcp_postgres_query is safer than query. Most hosts let you rename or prefix on the client side.
• Strip or delay the built-in tools that overlap with the servers you care about. Claude Code’s tool-search feature already defers some definitions; use it, or simply do not load the overlapping ones when you know you will need the MCP version.
• Keep the tool list short. Every extra schema is tokens and another chance for the model to pick the wrong one. Context Mode and similar sandboxes that summarize tool output exist for a reason.
• Measure the actual call distribution, not just the final answer. Log which tool the model chose on the first turn. If it is always the built-in, the MCP server is effectively unreachable.

# Quick way to see what tools a session actually sees
# (OpenCode / Claude Code both dump the tool list on demand)
opencode --debug 2>&1 | grep -A 20 '"tools"'
# or
claude --debug-tools

The same measurement works for any host that speaks MCP: dump the tools/list response and compare it to the functions that appear in the first model request.

Latency and the live path

The same principle shows up in real-time systems. A speech-to-text pipeline that has both a local Whisper instance and a cloud endpoint will call whichever one is listed first or described more strongly. On a Windows call-transcription path the capture layer itself is a harness decision: WASAPI loopback versus a mixed stream, exclusive mode versus shared mode, 16 kHz mono versus the device’s native format. Each of those choices changes the token stream the model sees and therefore which downstream tools it prefers.

A dual-track capture (microphone + system loopback) is one way to keep the two sides of a call separate so the STT sessions stay language-specific and the overlay can label speakers. The code that does the loopback is short:

// cpal + WASAPI loopback on a render endpoint
let device = find_output_device(name)?;
let default_cfg = device.default_output_config()?;
let config = StreamConfig {
    channels: default_cfg.channels(),
    sample_rate: default_cfg.sample_rate(),
    buffer_size: cpal::BufferSize::Default,
};
// build_input_stream on the output device yields the loopback

That capture path is the “built-in tool” of the audio stack. If your agent also has a higher-level “transcribe this file” tool that mixes everything together, the model will prefer the mixed path even when you attached a better dual-track server. The numbers from the MCP benchmark are the same numbers you will see in a live call if you do not control the tool list.

Trade-offs

A lighter harness scores higher on pure MCP-server tasks and costs fewer tokens. It also has fewer fallbacks when the server is down or rate-limited. A heavier harness (Claude Code style) is more resilient to missing servers but systematically under-uses the ones you add. Neither is free.

For a Windows desktop tool that must stay under a few hundred milliseconds of added latency, the practical choice is usually: keep the built-in surface tiny, expose only the tools you actually need for the current mode (interview overlay, translation, note-taking), and measure the first-tool-choice distribution on real sessions. The 17-point gap is not an anomaly; it is the expected result when two different tool lists are fed to the same weights.

I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.