In my first post about Jaeger's MCP server I made a claim I couldn't back up. The server tiers its nine tools by cost, cheap discovery first, verbose span data last, so that an agent drills down instead of dumping everything into its context. I measured the byte sizes and called it progressive disclosure. What I never measured was whether agents actually behave that way, or whether the whole tiered design beats the thing most teams build first: one tool that returns everything.
Meanwhile the internet is still running the MCP-token-cost argument I wrote about last time: strong claims, no controlled comparison.
So I ran the comparison. 72 trials, two tool designs, two model families, six trace troubleshooting tasks, pre-registered hypotheses. This is the second arm of jaeger-mcp-bench; the first arm, which varied a single tool's output format, is written up here.
The two arms
Tiered is the real thing: stock jaegertracing/jaeger:2.20.0 with ai.enable_mcp: true. That registers the nine shipped tools (get_services, search_traces, get_trace_topology, get_critical_path, get_span_details, get_trace_errors, and friends) plus the instructions file from post #1, nine lines at this release. Not a fork, not a patch. Anyone can pull the image and get this arm.
Flat is the naive integration, built for this benchmark: a small Go MCP server exposing exactly one tool, get_trace_data(service, lookback_minutes, limit, errors_only), which calls jaeger-query's /api/traces and returns every matching trace as a complete span dump. Every attribute, every log, every reference, verbatim, no summarization. It is the wrapper most people write on day one: expose the search endpoint, return the JSON, let the model figure it out.
Everything else is held constant. Same Jaeger backend, same frozen trace store, same streamable HTTP transport with identical session options, same per-trial budget, and deliberately neutral wording everywhere the model can see. The flat server's instructions open with the same one-line domain framing as the tiered server's and then stop. No "naive", no "baseline", no experiment vocabulary in anything model-visible, because a model that knows it's the control arm stops being a control arm.
The tasks are the kind of thing you'd actually ask: find the root cause of the errors, name the operation eating the critical path of the slowest trace, count the spans and services in the slowest trace, pull an attribute off a failing span, map who calls the database, explain where a slow request's extra time went. Ground truth is computed from the same HTTP API both arms read, using a Python port of Jaeger's own critical path algorithm, against a store frozen before the first trial. The fixture is volume-capped at 100 traces per service, which happens to be the tiered arm's server-side search cap, so both arms can enumerate the full candidate set. Whether they bother to is part of what's being tested.
One more thing before any trial ran: the design doc with hypotheses, task table, and exactly four statistical comparisons went into git first. The bar for "significant" was Bonferroni-corrected to 0.0125. Everything else in this post is labeled what it is: observation.
Correctness: the inventory decides, not the model
| Comparison (registered) | Correct | p | Significant? |
|---|---|---|---|
| Claude: tiered vs flat | 16/18 vs 6/18 | 0.0006 | yes |
| Gemini: tiered vs flat | 14/18 vs 8/18 | 0.04 | no (0.04 > 0.0125 corrected) |
| Tiered: Claude vs Gemini | 16/18 vs 14/18 | 0.37 | no |
| Flat: Claude vs Gemini | 6/18 vs 8/18 | 0.49 | no |
Pooled across both models, tiered solved 30 of 36 tasks and flat solved 14 of 36.
Correct here means the trial produced the right answer within its budget. A trial that exhausted budget or wall clock without answering counts as incorrect, because that is what it is in production. If you instead condition on trials that finished, flat Claude looks fine, which is exactly the accounting a fixed budget exists to prevent, and uncapped Gemini still got only 8 of 18 with genuinely wrong answers. The p values are two-proportion z-tests, per the registered plan; if you distrust the normal approximation at n=18, Fisher exact gives 0.0016 and 0.086 for the two tiered-vs-flat rows and changes no verdict.
Tiered did not sweep, either. On the trace-shape task Claude's tiered arm went 1 of 3 while flat went 2 of 3, and both of the tiered arm's budget deaths landed there. Six tasks, honest variance.
The result I keep looking at is the bottom half of the table: neither model cleared a significance bar against the other inside either arm, and at n=18 the test had little power to find one. What the observed numbers say is that swapping the model barely moved them and swapping the tool design doubled them. If you have a fixed budget to spend on making agents useful against your infrastructure, spend it on the tool design.
Cost: cap the naive arm and it fails, uncap it and it's 29x
The two CLIs disagreed about money in a useful way.
Claude ran with a $0.50 budget per trial, same as arm 1. Nine of its eighteen flat trials burned the entire budget without producing an answer, and two more hit the ten minute wall clock. The tiered arm lost two trials to budget in the same 18. Under a cap, the flat design doesn't get to be expensive. It just fails.
Gemini ran uncapped, and that shows you the bill the cap was hiding: the flat arm averaged 289,000 input tokens per trial against the tiered arm's 9,900. Twenty-nine times. The single worst trial pushed 2.0 million raw input tokens through the model across seven calls to answer one question about one error (the arm means above are cache-normalized; raw cumulative is the bill your provider sends). The tiered arm answered the same question correctly on about 13,000.
Same problem, two ways to pay for it. Give the everything-tool a budget and it exhausts it. Remove the budget and it exhausts you.
Behavior: they drill down, and mostly don't need the depths
The claim at the center of the server's design is behavioral: given tiered tools, an agent will fetch structure before detail. I captured full tool-call trajectories for six supplementary tiered trials, one per task, run after the scored matrix against the same frozen store, and classified each: did a discovery call precede the first verbose call?
Zero violations. Two trajectories drilled down exactly as designed, structure first, get_trace_errors only after. The other four never made a verbose call at all. Topology, trace summaries, and span names were enough to answer the question. That's the strongest version of the result: for most of these tasks the expensive tier isn't just deferred, it's unnecessary. Two of the six also called read_skill unprompted and pulled the embedded troubleshooting playbook before touching trace data, which I did not expect.
Six trajectories is a small sample and I report it as one. But zero-for-six violations, on the exact behavior the instructions file asks for, is not nothing.
The two traps the losers fell into
The error filter doesn't mean what agents think it means. Jaeger's error=true search matches error spans belonging to the queried service only. Ask for failing traces of the driver service and you get nothing, because the actual error status lives on redis-manual spans inside driver's traces. Both Gemini arms hit this, trusted the empty result, and reported "no failing traces in the last 24 hours" for a service that was failing the whole time. Claude hit the same empty results, kept digging without the filter, and got the answer anyway. This one is going upstream: it's not a bug, but it's semantics that reliably mislead an agent, and now there's trial data showing it.
Defaults silently truncate. The flat tool's limit defaults to 20, a choice I made building it; jaeger-query's own API defaults to 100, which would have covered this entire fixture. The schema discloses the default. Gemini fetched it anyway, took the slowest trace of that sample, and analyzed the wrong trace with total confidence, twice. The complete set was one parameter away. Nothing in a raw dump tells you it's incomplete; a tiered summary tool that returns total_count and truncated fields does. That's post #1's schema lesson wearing a different hat.
My harness lied to me before the models could
Confession section, as usual. The first full Claude run came back looking plausible and was garbage: a network blip during the run made the CLI return "API Error: Unable to connect" as if it were the model's answer, exit code zero, eighteen times. Every one of those would have been scored as the model getting the question wrong. I caught it because a 10-minute checkpoint probe showed a suspiciously fast, suspiciously uniform failure pattern. That run is quarantined in the repo with a note, the runner now detects and retries transport failures, and infrastructure errors are excluded from every denominator.
The scorer needed three rounds of the same medicine. Reading real answers against verdicts caught it penalizing exemplary responses, ones that named non-callers precisely to exclude them ("no other service calls mysql directly"), and being fooled by the word "route" being both a service and a verb. Each false verdict is now a pinned regression test; the suite is at 116.
This is the same lesson as the soak test that almost passed on an idle box: the instrument fails silently, the failure reads as a result, and the only defense is auditing the instrument against reality before believing it. A benchmark's biggest threat isn't the model gaming it. It's your own harness being politely wrong.
What this does not show
The comparison bundles tool inventory with serialization format. The flat arm returns jaeger-query's classic JSON, which is chattier per attribute than the tiered tools' output structs, so this measures "the shipped tiered design versus a naive wrapper as people actually build one", not tool-count in isolation. Arm 1 isolated format on its own; I didn't re-isolate it here, and the design doc says so.
Hotrod traces are small, tens of spans. At production sizes the flat arm's context pressure gets worse, not better, but that's extrapolation and I'm not claiming it. Six trajectories is descriptive, not inferential. Gemini's failure detection is heuristic because its CLI has no budget cap analogue. And n=18 per cell is honest for proportions and thin for everything subtler; the 18 are 6 tasks times 3 repeats, so they cluster by task rather than being 18 independent draws.
Takeaways
- Tool inventory design moved correctness more than model choice did. 83% vs 39% between designs, no detectable model effect inside either arm at this n.
- The everything-tool makes you pick your failure mode: budget-capped it fails to answer, uncapped it costs 29x.
- Agents given tiered tools used them as tiered. Most tasks never needed the verbose tier at all.
- Empty results are ambiguous to agents. If your filter has surprising scoping, the agent will trust the empty set and report a confident wrong answer. Say what the filter means in the schema.
- Audit your harness like it's hostile. Mine dressed a network outage as eighteen model failures and had exit code zero while doing it.
Everything is in the repo: pre-registered design with its amendment history in git, the flat server, the harness, raw trials including the quarantined run, and the tables. The tiered arm is a stock Jaeger image; reproducing this costs a docker pull, a seed, and API credits.
The design I couldn't back up in my first post is now the measured one. That was the point of writing the claim down before testing it.
I do this work for hire: benchmarking what your MCP tools actually cost agents, and building tool surfaces that hold up under measurement. Scope and pricing at roshansingh.systems/#hire, or write to inbox@roshansingh.systems.
Top comments (0)