DEV Community: Roshan Singh

I measured what 14 MCP servers cost a context window. Claude counts them 64% higher than tiktoken

Roshan Singh — Tue, 18 Aug 2026 17:11:54 +0000

Last month I ran 72 trials to settle what an MCP tool should return, because a maintainer would not take opinion for an answer. That left the other half open: before an agent does any work, how much of its context window have the attached servers eaten?

Vendors published numbers for this in 2026. I checked the six that get cited: exactly one is a real measurement study, StackOne's from 2026-03-31, which measured GitHub and Atlassian and worked through Cloudflare's code-mode case. Of the other five, one uses hypothetical unnamed servers, three contain no server-specific token study at all, and one I could not locate at any domain. So the published state of the art is a single snapshot, and it still shows March's numbers. A number that is not re-measured is a screenshot, and MCP servers change every few weeks. So I built a standing measurement, called loadline: 14 servers, methodology 0.2.0, one run dated 2026-08-18, monthly from here, every row reproducible from the repo with your own credentials. The front door is a stack calculator, not a table: the useful question is what my stack costs in the client I use.

Claude counts the same schema bytes around 64 percent higher than tiktoken

All three count one fixed string, the canonical serialization from methodology section 1.5. Same bytes, three counters.

server	o200k_base	claude-opus-5	premium
filesystem (14 tools)	2,697	4,432	+64.3%
linear (53 tools)	17,780	29,026	+63.3%
kubernetes (20 tools)	4,430	7,552	+70.5%
github (47 tools)	59,084	86,843	+47.0%

Across the 12 rows that produced counts the premium runs from 47.0 percent (github) to 70.5 percent (kubernetes), median 64.1, with 8 of the 12 between 60 and 66. Gemini stays within 11 percent of o200k on every row, so this is not a bigger-models-count-more effect. It is specific to Claude's tokenizer on schema text.

Every MCP cost study I have seen counts with tiktoken, the only one of the three that runs offline and free. If your agent runs on Claude, those studies describe a context load roughly 60 percent smaller than the one you pay for on the same schemas. They are not wrong about the bytes. They count them for a different model than the one in your loop.

GitHub MCP, loaded naively, is 43 percent of a 200k window

The github row measures 47 tools at 59,084 tokens under o200k_base and 86,843 under claude-opus-5. That second figure is 43 percent of a 200k window, spent before the first user message.

Two qualifiers matter more than the number. First, 47 tools is what the server exposed to the classic personal access token this run used, and the spec lets a surface vary with the authorization presented, so the real GitHub surface is larger. Auth scope publishes per row, since that is the largest source of legitimate disagreement here.

Second, nobody on a tool-search client pays it. Claude Code has defaulted to progressive disclosure since January 2026, and on this surface that models out to roughly 4,300 to 6,800 tokens: a 500-token stub plus 3 to 5 tools at the measured 1,257-token average. A swing that size, driven entirely by the client, is why no row publishes one cost number. Three modes, always. And "modeled" is load-bearing there, which I get to below.

Three tools can cover a product line

Cloudflare's aggregate endpoint answers tools/list with three tools: docs (362 tokens), search (572), execute (660). Total 1,596 under o200k, 37 times smaller than github's naive load, and hygiene holds at B (85.19), so the compression did not come from deleting descriptions.

The qualifier: that is the aggregate endpoint measured as one server. Cloudflare also ships 16 product-scoped endpoints this run did not enumerate or sum.

Chrome DevTools ships 52 tools without the token bill you would expect

Chrome DevTools MCP, official from Google and newly measured this run, exposes 52 tools, second only to linear's 53, at 7,984 tokens under o200k: 153 tokens per tool, the leanest average in the corpus. Playwright's 24 tools cost 167 each, filesystem's 14 cost 192, linear's 53 cost 335. Hygiene holds at B (80.28), so, as with Cloudflare, the leanness is not from deleting descriptions. A server can carry more surface than all but one other in the corpus and still land near the cheap end of it.

Two of the 14 servers could not start, and one of them healed an hour later

fetch, the reference implementation, and postgres both failed to launch from a clean install when the run executed, 2026-08-18 at 15:38 UTC. Each declares an unbounded dependency on the MCP Python SDK, mcp>=1.1.3 for fetch and mcp[cli]>=1.5.0 for postgres, so both take whatever the resolver hands them. A resolve that lands on mcp 2.0.0 breaks both, in two different places.

fetch imports a symbol that 2.0.0 renamed:

ImportError: cannot import name 'McpError' from 'mcp.shared.exceptions'.
Did you mean: 'MCPError'?

postgres dies one import earlier, on a module 2.0.0 no longer ships at that path:

ModuleNotFoundError: No module named 'mcp.server.fastmcp'

Then, re-checking both rows about an hour later on the same machine with the resolver cache forced fresh, fetch came back. A clean resolve handed it mcp 1.29.0 instead of 2.0.0, and it started. Nothing upstream was withdrawn: 2.0.0 is still the latest release and is not yanked. The same command, on the same machine, on the same afternoon, resolved differently. postgres still fails.

So the fetch: unreachable row this post is built on went stale about an hour after it was written, and I am publishing it that way, labelled, with an entry in the corrections log, rather than quietly regenerating the dataset until it agrees with the prose. Next month's run will say whatever is true next month.

The more useful outcome is that it found a hole in my own instrument. The harness records whether the server package was pinned, but it never records which SDK version the resolve actually produced, so the artifact cannot explain why 15:38 and 16:45 disagree. That gets fixed before the next run.

I could have pinned around the breakage in a minute and decided not to. Pinning changes what "the fetch server" means here, publishing a constrained old version while implying it describes what you would install today. Server rot is part of the subject matter, so it is part of the dataset.

figma does not publish as a row at all this run: it failed gate 2 of the published selection rule, full surface enumerable under a free-tier credential, because its auth is OAuth-only and a reproducible harness has no credential to exercise that flow. It was excluded from the corpus rather than published as an auth row, and chrome-devtools was promoted to the freed slot.

Cheap and findable are different axes

Notion's redesigned server is cheap: 24 tools, 5,180 tokens under o200k. It also scores hygiene D (54.79) and a top-3 retrievability fraction of 0.5417, the lowest measured here, with an MRR of 0.4527. What sinks it: when_to_use_signal 0, disambiguation 0, parameter_descriptions 44.03. Context7, at 2 tools and 1,052 tokens, scores hygiene A (97.22), top-3 1.0, MRR 1.0.

Under a naive client that difference is invisible: both surfaces are just text in the prompt. Under a tool-search client it is the whole game, because a tool the search does not surface is a tool the agent does not have, however cheap it is. Rank on schema size alone and the winning move is gutting descriptions.

What is measured and what is not

Every cell carries a MEASURED or MODELED label. Naive full-load is measured: the token count of the canonical serialization. The per-tool costs in the tool-search column are measured, but its total is modeled, because k, the number of tools a session pulls, is an assumption, which is why it publishes as a 3-to-5 range and not a point. Code mode is modeled end to end and keeps that label until a Tier 2 run validates it.

Retrievability queries are derived from each tool's own description, so the metric measures within-server disambiguation, whether sibling tools shadow each other, not whether real phrasing finds a tool. It shares input text with the hygiene grade, so the two are not independent evidence.

Dollar figures are estimates, computed from a price table pending verification and shown only as a cold-write and cache-read pair. Tokens stay the primary unit: a single dollar figure is the cold-start number presented as though it recurs, overstating steady state by roughly 10x, since tool definitions are a near-perfect cache prefix. The rest is in docs/methodology-v0.md 0.2.0, with the rejected alternative for each judgment call.

The governance, as mechanism

Neutrality is not something to take on my word, so here are the checkable parts.

Selection runs on a published rule: five binary gates (protocol compliance, full surface enumerable under a free-tier credential, verifiable adoption evidence, a release in the trailing six months, no official successor), plus a category spread and a 15-slot ceiling. Coverage is not a claim this project makes. Anyone can submit a server through a public issue and gets the gate result back, pass or fail, with the failing gate named.

The corrections log is public and append-only, and it has two entries before the first ranking: a mismatch between what the methodology said the harness did and what it did, caught in review before any release, and the fetch row above. Before a server's numbers first publish, its maintainer gets 14 days with the rows, the artifacts, and the methodology version; a reply publishes verbatim beside the row, and non-response is recorded as "no reply received", not as agreement.

Every row ships three SHA-256 hashes of the tool surface plus the raw wire artifacts, so a disputed count is checked against the bytes it came from, not against my word. Recusal is published before the first ranking, and my own project is excluded from rankings by rule.

What lands next

Repo: github.com/lopster568/loadline. Calculator: loadline-dev.netlify.app.

The next run adds slack, once its operator credential lands, and starts Tier 2: scripted task runs through a call-logging proxy, measuring real call and response flows. Tier 2 is the capability axis, and if it stops running the leaderboard pauses, because a cost-only ranking is worse than none.

If a number here is wrong, tell me and bring the row. The harness, corpus file, artifacts, and derived query sets are in the repo, so disagreement can be specific, and corrections go in the log with your name on them.

lopster568 / loadline

loadline

STATUS: pre-release draft. Not published. All text pending owner review.

loadline is a standing, versioned measurement of what MCP servers cost an agent's context window. It measures on a monthly cadence, run manually today per docs/pc-sweep-runbook.md, across a curated set of servers, counts tool-schema tokens with three tokenizer adapters, and reports the result through a stack calculator: pick servers, a client mode, and a model, and get total context footprint, window share, per-server attribution, and a cold-write/cache-read dollar pair. Published rows carry counts from all three tokenizers (OpenAI o200k locally, Claude and Gemini via their token-counting APIs); a cell whose count could not be obtained publishes as available: false, never as an estimate. Costs are always reported per client mode (naive full-load, tool search / progressive disclosure, code mode), never collapsed into one number because the same stack costs a different amount depending on how the client…

View on GitHub

I do this work for hire: auditing what your agent's tool surface costs before it does any work, and cutting it down without gutting what the search can still find. Scope and pricing at roshansingh.systems/#hire, or write to inbox@roshansingh.systems and tell me what your agents are loading.

Does progressive disclosure actually work? I benchmarked Jaeger's MCP server against the naive alternative

Roshan Singh — Mon, 10 Aug 2026 12:25:41 +0000

In my first post about Jaeger's MCP server I made a claim I couldn't back up. The server tiers its nine tools by cost, cheap discovery first, verbose span data last, so that an agent drills down instead of dumping everything into its context. I measured the byte sizes and called it progressive disclosure. What I never measured was whether agents actually behave that way, or whether the whole tiered design beats the thing most teams build first: one tool that returns everything.

Meanwhile the internet is still running the MCP-token-cost argument I wrote about last time: strong claims, no controlled comparison.

So I ran the comparison. 72 trials, two tool designs, two model families, six trace troubleshooting tasks, pre-registered hypotheses. This is the second arm of jaeger-mcp-bench; the first arm, which varied a single tool's output format, is written up here.

The two arms

Tiered is the real thing: stock jaegertracing/jaeger:2.20.0 with ai.enable_mcp: true. That registers the nine shipped tools (get_services, search_traces, get_trace_topology, get_critical_path, get_span_details, get_trace_errors, and friends) plus the instructions file from post #1, nine lines at this release. Not a fork, not a patch. Anyone can pull the image and get this arm.

Flat is the naive integration, built for this benchmark: a small Go MCP server exposing exactly one tool, get_trace_data(service, lookback_minutes, limit, errors_only), which calls jaeger-query's /api/traces and returns every matching trace as a complete span dump. Every attribute, every log, every reference, verbatim, no summarization. It is the wrapper most people write on day one: expose the search endpoint, return the JSON, let the model figure it out.

Everything else is held constant. Same Jaeger backend, same frozen trace store, same streamable HTTP transport with identical session options, same per-trial budget, and deliberately neutral wording everywhere the model can see. The flat server's instructions open with the same one-line domain framing as the tiered server's and then stop. No "naive", no "baseline", no experiment vocabulary in anything model-visible, because a model that knows it's the control arm stops being a control arm.

The tasks are the kind of thing you'd actually ask: find the root cause of the errors, name the operation eating the critical path of the slowest trace, count the spans and services in the slowest trace, pull an attribute off a failing span, map who calls the database, explain where a slow request's extra time went. Ground truth is computed from the same HTTP API both arms read, using a Python port of Jaeger's own critical path algorithm, against a store frozen before the first trial. The fixture is volume-capped at 100 traces per service, which happens to be the tiered arm's server-side search cap, so both arms can enumerate the full candidate set. Whether they bother to is part of what's being tested.

One more thing before any trial ran: the design doc with hypotheses, task table, and exactly four statistical comparisons went into git first. The bar for "significant" was Bonferroni-corrected to 0.0125. Everything else in this post is labeled what it is: observation.

Correctness: the inventory decides, not the model

Comparison (registered)	Correct	p	Significant?
Claude: tiered vs flat	16/18 vs 6/18	0.0006	yes
Gemini: tiered vs flat	14/18 vs 8/18	0.04	no (0.04 > 0.0125 corrected)
Tiered: Claude vs Gemini	16/18 vs 14/18	0.37	no
Flat: Claude vs Gemini	6/18 vs 8/18	0.49	no

Pooled across both models, tiered solved 30 of 36 tasks and flat solved 14 of 36.

Correct here means the trial produced the right answer within its budget. A trial that exhausted budget or wall clock without answering counts as incorrect, because that is what it is in production. If you instead condition on trials that finished, flat Claude looks fine, which is exactly the accounting a fixed budget exists to prevent, and uncapped Gemini still got only 8 of 18 with genuinely wrong answers. The p values are two-proportion z-tests, per the registered plan; if you distrust the normal approximation at n=18, Fisher exact gives 0.0016 and 0.086 for the two tiered-vs-flat rows and changes no verdict.

Tiered did not sweep, either. On the trace-shape task Claude's tiered arm went 1 of 3 while flat went 2 of 3, and both of the tiered arm's budget deaths landed there. Six tasks, honest variance.

The result I keep looking at is the bottom half of the table: neither model cleared a significance bar against the other inside either arm, and at n=18 the test had little power to find one. What the observed numbers say is that swapping the model barely moved them and swapping the tool design doubled them. If you have a fixed budget to spend on making agents useful against your infrastructure, spend it on the tool design.

Cost: cap the naive arm and it fails, uncap it and it's 29x

The two CLIs disagreed about money in a useful way.

Claude ran with a $0.50 budget per trial, same as arm 1. Nine of its eighteen flat trials burned the entire budget without producing an answer, and two more hit the ten minute wall clock. The tiered arm lost two trials to budget in the same 18. Under a cap, the flat design doesn't get to be expensive. It just fails.

Gemini ran uncapped, and that shows you the bill the cap was hiding: the flat arm averaged 289,000 input tokens per trial against the tiered arm's 9,900. Twenty-nine times. The single worst trial pushed 2.0 million raw input tokens through the model across seven calls to answer one question about one error (the arm means above are cache-normalized; raw cumulative is the bill your provider sends). The tiered arm answered the same question correctly on about 13,000.

Same problem, two ways to pay for it. Give the everything-tool a budget and it exhausts it. Remove the budget and it exhausts you.

Behavior: they drill down, and mostly don't need the depths

The claim at the center of the server's design is behavioral: given tiered tools, an agent will fetch structure before detail. I captured full tool-call trajectories for six supplementary tiered trials, one per task, run after the scored matrix against the same frozen store, and classified each: did a discovery call precede the first verbose call?

Zero violations. Two trajectories drilled down exactly as designed, structure first, get_trace_errors only after. The other four never made a verbose call at all. Topology, trace summaries, and span names were enough to answer the question. That's the strongest version of the result: for most of these tasks the expensive tier isn't just deferred, it's unnecessary. Two of the six also called read_skill unprompted and pulled the embedded troubleshooting playbook before touching trace data, which I did not expect.

Six trajectories is a small sample and I report it as one. But zero-for-six violations, on the exact behavior the instructions file asks for, is not nothing.

The two traps the losers fell into

The error filter doesn't mean what the schema says it means. Jaeger's error=true search matches error spans belonging to the queried service only. Ask for failing traces of the driver service and you get nothing, because the actual error status lives on redis-manual spans inside driver's traces. I have exactly one clean measurement of this, from a captured trajectory: the same query with with_errors: true returned an empty result, and without it returned 2,882 bytes including a trace that did contain error spans. That agent recovered by dropping the filter. Five of six Gemini trials on this task also answered "no failing traces", but that harness logged tool names without arguments, so I can't show the filter caused any of them, and three of the five never called search at all. One measurement, reported as one.

The upstream part is worth more than the trap. Jaeger's MCP schema documents the flag as "If true only return traces containing error spans", which is trace-scoped language for span-scoped behavior. My own flat server describes it correctly, because I hit the confusion while building it and wrote the caveat into the tool description. That's the fixable thing, and it's one line of documentation, not a design argument.

Defaults silently truncate. The flat tool's limit defaults to 20, a choice I made building it; jaeger-query's own API defaults to 100, which would have covered this entire fixture. The schema discloses the default. Gemini fetched it anyway, took the slowest trace of that sample, and analyzed the wrong trace with total confidence, twice. The complete set was one parameter away. Nothing in a raw dump tells you it's incomplete; a tiered summary tool that returns total_count and truncated fields does. That's post #1's schema lesson wearing a different hat.

My harness lied to me before the models could

Confession section, as usual. The first full Claude run came back looking plausible and was garbage: a network blip during the run made the CLI return "API Error: Unable to connect" as if it were the model's answer, exit code zero, eighteen times. Every one of those would have been scored as the model getting the question wrong. I caught it because a 10-minute checkpoint probe showed a suspiciously fast, suspiciously uniform failure pattern. That run is quarantined in the repo with a note, the runner now detects and retries transport failures, and infrastructure errors are excluded from every denominator.

The scorer needed three rounds of the same medicine. Reading real answers against verdicts caught it penalizing exemplary responses, ones that named non-callers precisely to exclude them ("no other service calls mysql directly"), and being fooled by the word "route" being both a service and a verb. Each false verdict is now a pinned regression test; the suite is at 116.

This is the same lesson as the soak test that almost passed on an idle box: the instrument fails silently, the failure reads as a result, and the only defense is auditing the instrument against reality before believing it. A benchmark's biggest threat isn't the model gaming it. It's your own harness being politely wrong.

What this does not show

The comparison bundles tool inventory with serialization format. The flat arm returns jaeger-query's classic JSON, which is chattier per attribute than the tiered tools' output structs, so this measures "the shipped tiered design versus a naive wrapper as people actually build one", not tool-count in isolation. Arm 1 isolated format on its own; I didn't re-isolate it here, and the design doc says so.

Hotrod traces are small, tens of spans. At production sizes the flat arm's context pressure gets worse, not better, but that's extrapolation and I'm not claiming it. Six trajectories is descriptive, not inferential. Gemini's failure detection is heuristic because its CLI has no budget cap analogue. And n=18 per cell is honest for proportions and thin for everything subtler; the 18 are 6 tasks times 3 repeats, so they cluster by task rather than being 18 independent draws.

Takeaways

Tool inventory design moved correctness more than model choice did. 83% vs 39% between designs, no detectable model effect inside either arm at this n.
The everything-tool makes you pick your failure mode: budget-capped it fails to answer, uncapped it costs 29x.
Agents given tiered tools used them as tiered. Most tasks never needed the verbose tier at all.
Empty results are ambiguous to agents. If your filter has surprising scoping, the agent will trust the empty set and report a confident wrong answer. Say what the filter means in the schema.
Audit your harness like it's hostile. Mine dressed a network outage as eighteen model failures and had exit code zero while doing it.

Everything is in the repo: pre-registered design with its amendment history in git, the flat server, the harness, raw trials including the quarantined run, and the tables. The tiered arm is a stock Jaeger image; reproducing this costs a docker pull, a seed, and API credits.

The design I couldn't back up in my first post is now the measured one. That was the point of writing the claim down before testing it.

I do this work for hire: benchmarking what your MCP tools actually cost agents, and building tool surfaces that hold up under measurement. Scope and pricing at roshansingh.systems/#hire, or write to inbox@roshansingh.systems.

What should an MCP tool return? I ran 72 trials instead of arguing

Roshan Singh — Fri, 07 Aug 2026 17:41:39 +0000

There's an argument running about MCP right now. You've probably seen it: a 400-point thread called "MCP is dead?" with real token numbers in it, four connected servers eating 21,077 tokens of context before anyone asks a question. The argument is about what MCP costs. Almost nobody in it has measured what agents actually do with the data a tool returns.

I ended up measuring that, not because I planned to, but because a maintainer refused to let me guess.

The question nobody wanted to answer with opinions

I contribute to CNCF Jaeger's MCP server. Last April I proposed exposing service performance metrics (latencies, call rates, error rates) as an MCP tool, and hit an immediate design fork: what shape should the output be?

Option one, summary rows: pre-aggregated stats per service, compact, cheap. Option two, per-bucket time series: the raw points, roughly 720 of them per service at default resolution, expensive but complete.

I asked which the maintainer preferred. The answer, verbatim, from the issue thread:

This type of decision should not be based on opinion, but on benchmarks with a real agent troubleshooting some issues and using this MCP tool to access metrics, where you could do A/B testing of different output formats.

Fair. So I built the A/B.

The setup

Everything below is public in jaeger-mcp-bench, including the harness, the tasks, the scorer, and a research log of everything that went wrong.

The fixture is Jaeger v2 with the spanmetrics connector, hotrod generating traffic, and Prometheus behind it, snapshotted so every run sees identical metric state. In front of the metrics API sits a thin bench server with exactly one switch: --format=summary|series. No new semantics, just the shape of what comes back.

Six troubleshooting tasks, and this part matters: three were chosen because I predicted summary would win them (point questions: current latency, ranking, threshold checks) and three because I predicted series would win (temporal questions: spike detection, correlation, trend). Designing the tasks to favor both arms keeps the deck unstacked against your own hypothesis.

Two agents, and not stripped-down tool loops: Claude Sonnet through the Claude Code CLI and Gemini 2.5 Pro through the gemini CLI, real system prompts and all, because that's what production agents actually are. Three trials per cell, 72 trials total, cells run in randomized order (seed 42) so fixture drift can't correlate with either arm. Scoring is programmatic against ground truth, not vibes.

Note: Every trial ran headless: claude -p and the gemini CLI invoked one shot per trial, no session and no one to rephrase, so a decline ended the trial rather than starting a retry.

What 72 trials said

arm	correct	wrong	declined
claude / series	18	0	0
gemini / series	17	0	1
claude / summary	10	1	7
gemini / summary	11	0	7

The result I expected was wrong answers. Bad format, wrong conclusions, embarrassing agent behavior, good story.

That is not what happens. Across 72 trials there was exactly one wrong commitment, and it traced to a bug in my own bench server, not to the format (disclosed in RESULTS.md; it biased against summary, and I'd rather report my own bug than a fake finding).

The real difference lives in the decline rate. Given summary rows, agents said some version of "I cannot determine this from the available data" seven times more often than with series, almost entirely on the temporal questions. And they were right to decline: aggregation had destroyed the time axis the question needed. You cannot localize a spike in an average.

Agents don't get it wrong when you underfeed them. They give up, politely and correctly.

The stats hold up under correction: Claude's summary-versus-series gap is significant after Bonferroni (p=0.001 against a 0.0125 alpha). Gemini's, at p=0.016, sits right on the boundary, and I'd rather say that than round it in my favor. Point queries were a wash, exactly as predicted: when the question needs one number, format doesn't matter.

So series won. But that's not the useful lesson

The tool shipped with per-bucket series as the decision, backed by data instead of my taste. Fine.

The useful lesson is about what "expensive output" actually buys. The whole MCP cost argument treats tokens as waste: big responses bad, small responses good. The benchmark says the relationship is more specific than that. A compact format is cheap right up until it deletes the axis a question depends on, and then its cost is not tokens, it's the agent declining to answer at all. A "cannot determine" costs you the entire investigation loop, plus the retry, plus whatever the human does when the agent shrugs.

Put differently: the token bill of series output is visible and easy to complain about. The failure bill of summary output is invisible until you measure decline rates, and nobody measures decline rates.

If you're designing an MCP tool right now, the transferable version is:

Match the format to the question class, not to the token budget. Point questions tolerate aggregation. Temporal and causal questions do not.
Watch declines, not just errors. Models fail politely. Your error dashboards won't see it.
Balance your benchmark against your own prediction, or you'll build a deck stacked toward whatever you already believed.
When your bench produces a result you like, hunt for the bug first. Mine had one, and it flattered the winning arm.

Honest limits

Seventy-two trials is three per cell. Two model families, one fixture, six tasks, all in the observability domain. This settles which format a metrics tool in a tracing backend should return; it does not settle MCP philosophy. If you run the harness against other domains I'd genuinely like to see the numbers, and everything needed to do that is in the repo.

The maintainer was right, is the short version. The format decision took a weekend to benchmark and would have taken zero minutes to argue about forever.

I do this work for hire, mostly making LLM agents safe and observable against production infrastructure. Scope and pricing are at roshansingh.systems/#hire, or write to inbox@roshansingh.systems and tell me what your agents touch.

The Bloom filter that never existed, and the two ceilings it was hiding

Roshan Singh — Fri, 31 Jul 2026 09:41:30 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

The most expensive bug I fixed this year was not in the code. It was in the documentation, and it had been shaping what everyone believed the code did.

The setup

HydraDNS is an open-source DNS security gateway I build in Go. Router points at it, it filters every DNS query on the network against a 92k-domain blocklist, blocks the bad ones, forwards the rest. Before putting it on anyone else's network I wanted a real number for what one box could take, so I sat down with dnspyre and a rule I had written for myself: every number becomes a sales claim or a fix ticket. No number, no claim.

Our feature sheet said the blocklist was backed by a Bloom filter, sub-millisecond lookups. Here is the uncomfortable part: at every load this system had ever run, that claim was indistinguishable from the truth. Normal-traffic latency sat at one or two milliseconds. There was nothing to doubt, because nothing observable disagreed.

The first ceiling

The redline test capped at about 500 queries per second. Odd, but fine, until I noticed the cap would not move. Blocked queries capped at ~500. Cached queries that never touch upstream also capped at ~500. Two paths doing completely different work, same wall, CPU sitting under 30% on a 22-core dev machine.

That combination is worth memorizing: when two very different code paths hit the same ceiling and the CPU is bored, the bottleneck is not in either path. It is in something they share.

Ours was the blocklist check. IsBlocked ran a SQL COUNT against the 92k-row table on every single query, because the check sits in front of the cache, so even cache hits paid for it. Every one of those reads was serialized through a single SQLite connection, MaxOpenConns=1, which was also absorbing the async write traffic from query logging. Engine self-latency under load: p50 of 50ms, p99 of five full seconds. For DNS.

And the Bloom filter? I went looking for it so I could tune it, and found a misattribution rather than a fabrication. The codebase does have a Bloom filter, a real one with tests, in the policy engine next door. Somewhere across three documents, "the policy engine uses a Bloom filter" drifted into "the blocklist uses a Bloom filter" and started reading like established fact. Every sanity check, mine included, recognized a component that genuinely exists and moved on.

Here is what makes this class of bug nasty: reading the blocklist code would not have flagged it either. The SQL path was correct code doing exactly what it claimed. Code review asks whether an implementation is right, and it was. No review asks whether an implementation matches an adjective in a feature sheet. The only reviewer that can ask that question is a measurement at redline, and that is precisely why the stress plan existed.

The fix, which is embarrassingly boring

I built the thing the docs had been imagining, minus the cleverness: MemoryChecker, an atomic in-memory domain set, loaded at startup, swapped wholesale on the existing 6-hour blocklist refresh. The parent-domain walk (block example.com, catch ads.example.com) reproduces the exact candidate set the SQL version checked, so behavior stayed identical. Only the cost changed.

Throughput: ~500 to ~9,500 QPS. Nineteen times, from taking one membership check off the database. Not a probabilistic data structure. A map.

The second ceiling was waiting behind the first

Re-ran the redline. The box got fast, then started eating memory: RSS climbed from 98MiB to a little over 2GB as load went to 10,000 QPS. It drained back when load stopped, so not a leak, a backlog. But the target hardware for this thing has 2 to 4GB of RAM total. That transient is an OOM kill in the middle of a traffic burst, which is the exact moment a DNS server must not die.

Query logging spawned a goroutine per query, each doing an INSERT plus a stats UPDATE through that same single connection. At 500 QPS the queue drained and nobody noticed. At 10,000, goroutines piled up faster than one connection could ever clear them. The first bottleneck had been rationing the second. Lift one ceiling and the next bomb arms itself.

The replacement: one writer goroutine, a bounded channel of 4,096 entries, batches flushed at 256 entries or 500ms, bulk INSERT plus one aggregated stats UPDATE per batch. Enqueue is non-blocking, and when the buffer is full the entry is dropped and the drop is counted, because losing a log line is acceptable for a DNS server and stalling resolution is not.

Before and after

Throughput ceiling:   ~500  -> ~9,500 QPS   (19x)
Engine p99:           5000ms -> 20ms
RSS at 10k QPS:       2,063MiB -> ~85MiB
CPU at 10k QPS:       56% -> 14%

Confirmed with a soak: 248,166 queries at 1,379 QPS, zero engine errors, memory a bounded sawtooth instead of a climb. All numbers from a 22-core dev machine with load generated in-container, and I label them that way everywhere, because the other lesson of this story is what happens to unverified claims.

What I learned

Same ceiling on two different paths plus an idle CPU means stop optimizing the paths and find what they share.
Fixing one bottleneck unmasks the next. The box's real limit is wherever you stop looking.
Unbounded "async" is not async, it is an OOM with a delay on it. Bound every queue and count every drop.
The one I keep thinking about: some claims are empirically true at every load you have run and false at the load you have not. They survive code review, doc review, and months of correct operation, because nothing observable disagrees with them. The only experiment that can falsify them is a redline test, which is what a stress plan is actually for. Not big numbers for a landing page. Falsification.

The longer write-up, with the full latency data and the test harness bug that almost let the soak pass on an idle server, is here: https://dev.to/lopster568/two-ceilings-taking-a-go-dns-server-from-500-to-9500-qps-2poj

Two ceilings: taking a Go DNS server from 500 to 9,500 QPS

Roshan Singh — Wed, 29 Jul 2026 06:13:42 +0000

I run HydraDNS, an open-source DNS security gateway in Go. Last month I sat down to find out what one box could actually handle before I put it on anyone else's network. The plan had a rule I'd written for myself: every number we discover becomes either a sales claim or a fix ticket. No number, no claim.

I expected to find one bottleneck. I found two, stacked on top of each other, and a third thing I wasn't looking for: a data structure in our own documentation that had never existed in the code.

Everything below was measured on a 22-core dev machine with load generated inside the container, using dnspyre, so docker-proxy and host networking stay out of the numbers. It's not appliance hardware and I'm not making appliance claims. The shapes are what matter.

The first ceiling: ~500 QPS, and it didn't care what I threw at it

The first redline run capped at roughly 500 queries per second. Fine, servers have limits. What made it interesting was that the cap didn't move. Blocked queries: ~500. Cached queries that never touch upstream: ~500. Two code paths that do completely different work, hitting the same wall, with the CPU sitting under 30% of 22 cores.

That signature is worth memorizing. When two very different paths hit the same ceiling and the CPU is bored, the bottleneck isn't in either path. It's in something they share, or something upstream of both.

Ours was in the blocklist check. IsBlocked ran a SQL COUNT against a 92k-row blocklist_entries table on every query. Not just candidate blocks, every query, because the check sits in front of the cache, so even cache hits paid for it. And all of those reads were serialized through a single SQLite connection, MaxOpenConns=1, which was also absorbing the async write traffic from query logging.

The engine's self-measured latency under load: p50 of 50ms, p99 of 5000ms. Five full seconds at the tail, for DNS, which is supposed to be the fast part of the internet.

The part where I found out our docs were lying

Here's the uncomfortable bit. Our feature sheet, and two other internal docs, said the blocklist was backed by a Bloom filter, sub-millisecond membership checks. And at every load this system had ever run, that claim was indistinguishable from the truth: normal-traffic latency sat at one or two milliseconds. Nothing observable disagreed with it.

What I found was a misattribution, not a fabrication. The codebase does have a Bloom filter, a real one with tests, in the policy engine next door. Somewhere across three documents, "the policy engine uses a Bloom filter" drifted into "the blocklist uses a Bloom filter," and a claim about the wrong component started reading like established fact. Every sanity check, mine included, recognized a component that genuinely exists and moved on.

The nasty part is that reading the blocklist code would not have caught it either. The SQL path was correct code doing exactly what it claimed, and code review asks whether an implementation is right, not whether it matches an adjective in a feature sheet. The only reviewer that can ask that question is a measurement at redline, which is exactly what this stress test was for. It forced me to look at what the hot path actually did, and what it did was a table scan's worth of SQL per packet.

So the fix wasn't "tune the Bloom filter." The fix was to build the in-memory layer the docs had been imagining. MemoryChecker: an atomic in-memory domain set, loaded at startup, swapped wholesale on the existing 6-hour blocklist refresh. The parent-domain walk (block example.com, and ads.example.com matches too) reproduces exactly the candidate set the SQL version checked, so behavior didn't change, just the cost.

Throughput went from ~500 to ~9,500 QPS. Nineteen times, from moving one membership check off the database. Not a clever data structure. A map, held in memory, reloaded every six hours.

The second ceiling was hiding behind the first

With reads fixed, I re-ran the redline, and the box got fast and then started eating memory. RSS climbed from 98MiB to a bit over 2GB as offered load went from 500 to 10,000 QPS.

It drained back to normal when the load stopped, so it wasn't a leak, it was a backlog. But a 2GB transient spike is academic only until you remember the target hardware has 2 to 4GB of RAM total. On the appliance this is an OOM kill in the middle of a traffic burst, which is the exact moment you don't want your DNS to die.

The cause: query logging spawned a goroutine per query, and each one did an INSERT plus a stats UPDATE through that same single SQLite connection. At 500 QPS the queue drained fine and nobody noticed. At 10,000 QPS, goroutines piled up faster than one connection could ever clear them. The first bottleneck had been rationing the second one. Lift the read ceiling and the write bomb goes off.

"Async logging" is only safe if it's bounded. Unbounded goroutine-per-event isn't async, it's a memory spike with extra steps, and it just moves the failure from slow to OOM.

The replacement is querylog_writer.go: one writer goroutine, a bounded channel of 4,096 entries, batches flushed at 256 entries or 500ms, bulk INSERT plus a single aggregated stats UPDATE per batch. Enqueue is non-blocking. If the buffer is full, the entry is dropped and the drop is counted, because losing a log line is an acceptable failure for a DNS server and stalling resolution is not.

After: RSS flat at ~85MiB from 500 to 10,000 QPS. Engine latency p50 5ms, p99 20ms. CPU at 10k QPS dropped from 56% to 14%, because it turns out scheduling tens of thousands of goroutines was itself a workload.

The numbers, before and after

Throughput ceiling:   ~500  -> ~9,500 QPS   (19x)
Engine p99:           5000ms -> 20ms
RSS at 10k QPS:       2,063MiB -> ~85MiB
CPU at 10k QPS:       56% -> 14%

Then a 3-minute soak to confirm the shape holds: 248,166 queries at 1,379 QPS, zero engine errors, RSS a bounded sawtooth (119 to 139 to 114MiB as GC does its thing) instead of the old monotonic climb.

The soak that almost passed on an idle box

One more confession, because it's the most useful lesson in here. My first soak attempt reported beautifully. Flat memory, no errors, perfect.

It had also sent zero queries. Deploying the retention change had recreated the container, which wiped the dnspyre binary inside it, and the docker exec failures were being swallowed by a 2>/dev/null I'd added earlier. The harness "ran," measured an idle server, and called it healthy. I caught it only because the sent-count looked suspiciously round: zero.

If your load generator's failure mode is silence, your benchmark's failure mode is a false pass. Now the harness asserts the sent count before it's allowed to report anything.

What I'd take to your codebase

Same ceiling on two different paths plus idle CPU means the bottleneck is shared or upstream. Stop optimizing the paths.
Fixing one bottleneck unmasks the next. The box's real limit is wherever you stop looking.
Bound every async path. A queue without a cap is an OOM with a delay on it.
Dropping data beats stalling the hot path, if you count the drops.
Some claims are true at every load you have run and false at the load you have not. They survive code review and months of correct operation because nothing observable disagrees with them. The only experiment that falsifies them is a redline test. That is what a stress plan is for.

That last one is the one I keep thinking about. The expensive bug wasn't in the code. It was in the docs, quietly shaping what everyone believed the code did.

I do this work for hire, mostly making LLM agents safe and observable against production infrastructure, and occasionally chasing a database off a hot path. Scope and pricing are at roshansingh.systems/#hire, or write to inbox@roshansingh.systems.

Building Jaeger's MCP server: connecting LLMs to distributed traces

Roshan Singh — Tue, 28 Jul 2026 13:40:00 +0000

A distributed trace is one of the few places where a system tells you the truth about itself. It records what actually called what, in what order, and how long each hop took. It is also, for a human being at 2am, a wall of a thousand spans.

That combination makes tracing data an unusually good target for an LLM. The data is structured, causal, and too large to read by hand. Over the last few months I shipped seven merged PRs into CNCF Jaeger's MCP server, the component that lets an LLM query traces as a first-class tool rather than by scraping the UI.

This post is about how that server is built in Go, and about the one design lesson that took me three PRs and a maintainer's review to actually internalize.

What the server is

The Model Context Protocol (MCP) is a way to hand an LLM a typed set of tools with schemas, instead of an API and hope. Jaeger's implementation lives in the main Go binary as a package that wraps the existing QueryService:

// Package mcptools provides the Jaeger telemetry MCP tools as a reusable
// library. The tools wrap a *querysvc.QueryService.
package mcptools

func NewServer(telset telemetry.Settings, queryAPI *querysvc.QueryService, cfg Config) *mcp.Server {
    server := mcp.NewServer(
        &mcp.Implementation{
            Name:    cfg.ServerName,
            Version: cfg.ServerVersion,
        },
        &mcp.ServerOptions{
            Instructions: serverInstructions,
        },
    )
    registerTools(server, queryAPI, cfg)
    // ...
    server.AddReceivingMiddleware(mw...)
    return server
}

Three things are worth pulling out of that.

It takes a *querysvc.QueryService directly, not a storage handle. The MCP tools do not reach into Cassandra or Elasticsearch. They go through the same query layer the Jaeger UI uses, which means tenancy, storage backends, and query semantics are all inherited for free. This was a maintainer correction early on: the original ADR had the extension depending on jaegerstorage, and it was changed to depend on the query service instead. Wrapping the layer that already has the business logic is almost always the right call for an MCP server.

Instructions are embedded, not configured. serverInstructions is a //go:embed INSTRUCTIONS.md string handed to ServerOptions.Instructions, which the SDK returns during the MCP initialize handshake. The LLM client receives the server's usage guidance automatically. Nobody has to paste anything into a config file.

Middleware is the observability seam. The server registers receiving middleware for tracing and metrics. There is a pleasing recursion in a tracing backend emitting traces about the tool calls an agent makes against it, and it turns out to be genuinely useful for debugging agent behavior.

The transport is streamable HTTP, wrapped with tenancy extraction and otelhttp, and mounted on an existing mux rather than binding its own listener:

func WrapHTTP(server *mcp.Server, tenancyMgr *tenancy.Manager, telset telemetry.Settings) http.Handler {
    mcpHandler := mcp.NewStreamableHTTPHandler(
        func(*http.Request) *mcp.Server { return server },
        &mcp.StreamableHTTPOptions{
            JSONResponse:   false, // Use SSE for streamed events
            Stateless:      false, // Session state management
            SessionTimeout: mcpSessionTimeout,
        },
    )
    tenantHandler := tenancy.ExtractTenantHTTPHandler(tenancyMgr, mcpHandler)
    return otelhttp.NewHandler(
        tenantHandler,
        "jaeger_mcp",
        otelhttp.WithTracerProvider(telset.TracerProvider),
    )
}

One server instance is reused across sessions. With Stateless: false the SDK builds one ServerSession per MCP session and reuses it for that session's requests.

The tools, and why they are shaped that way

The server currently registers nine tools. The interesting part is not the list, it is that the list is deliberately tiered by cost:

get_services, get_span_names - cheap discovery
search_traces - lightweight summaries, no spans or attributes
get_trace_topology - trace structure as a flat depth-first list, no attributes
get_critical_path - the chain of spans that determined end-to-end duration
get_trace_errors, get_span_details - verbose, full OTLP data
get_service_dependencies - the service graph
read_skill - built-in analysis playbooks

This is progressive disclosure, and it exists for one reason: context window economy. A topology span is roughly 185 bytes. A full span detail is roughly 1KB. When I measured that during the instructions work, the ratio was about 6x. An agent that calls get_span_details on every span in a 500-span trace burns its entire context before it has formed a hypothesis. An agent that starts with topology, finds the critical path, and then fetches details for four spans, answers the question.

So the whole design problem becomes: how do you get the LLM to drill down instead of dumping?

My first answer was wrong.

The lesson: the schema is the interface, not the prompt

I opened PR #8253 to add an INSTRUCTIONS.md system prompt. My draft was thorough in the way that draft documentation is thorough: a tool reference table, investigation patterns, a constraints section, a system limits section. It explained the drill-down order carefully. It was, I thought, the fix.

The maintainer review pushed back on the size, and I trimmed it. Then came the comment that actually mattered:

we could improve some tool descriptions to be more explicit about the limits and truncation. For example [...] The ErrorCount could be renamed to TotalErrorCount and the Spans list could say (possibly truncated to MaxXyz size)

That reframed the problem. I had been writing a document about the tools, sitting outside them, that an agent may or may not have in context by the time it decides what to call. The information belonged in the tools, in the schema, at the exact point of decision.

INSTRUCTIONS.md shipped at five lines. Here is essentially all of it:

Jaeger is a distributed tracing backend. A trace is a tree of spans representing a request or workflow [...] These tools support progressive disclosure to manage context density. While they can be called in any order based on available data, prefer starting with broad discovery (get_services or search_traces) or structural overviews (get_trace_topology) before requesting verbose OTLP details for specific spans.

Everything else moved into the schemas. The trace errors output went from a bare ErrorCount to this:

type GetTraceErrorsOutput struct {
    TraceID         string       `json:"trace_id" jsonschema:"Unique identifier for the trace"`
    TotalErrorCount int          `json:"total_error_count" jsonschema:"Total number of error spans in the trace (may exceed the size of the spans list due to per-request limits)"`
    Spans           []SpanDetail `json:"spans,omitempty" jsonschema:"Error span details (possibly truncated to server-configured limit)"`
}

Read that as an agent. total_error_count tells you there were 40 errors. The spans schema tells you the list you are holding may be truncated. The agent can now detect its own truncation by comparing the two numbers, without being told to in a prompt it might not be attending to.

I followed up in PR #8314 by rewriting all the tool descriptions on the same principle. Each one now states what the tool does, what distinguishes its output, and the runtime behavior the schema cannot express:

mcp.AddTool(s.mcpServer, &mcp.Tool{
    Name: "get_trace_topology",
    Description: "Get the structural overview of a trace as a flat, depth-first span list. " +
        "Each span includes a 'path' field encoding ancestry as slash-delimited span IDs " +
        "(e.g. 'rootID/parentID/spanID'). " +
        "Does NOT include attributes, events, or links.",
}, handlers.NewGetTraceTopologyHandler(s.queryAPI, s.config.MaxSpanDetailsPerRequest))

Note what is not there: no workflow ordering, no "call this before that." The input and output schemas already encode the dependencies. Sequencing instructions in a description are a smell that your schema is underspecified.

The generalizable rule: a tool description is read at the moment of the decision it governs. A system prompt is read once, at the start, and competes with everything since. Put the guidance where the decision happens.

Adding a tool, end to end

get_service_dependencies (PR #8403) is a clean example of the whole shape. QueryService.GetDependencies() was the only query method with no MCP tool, so an agent asking "what does payment-service call?" had to fetch traces and rebuild the graph by hand, even though Jaeger already computes and stores it.

The types come first, because in MCP the types are the API:

type GetDependenciesInput struct {
    StartTime string `json:"start_time,omitempty" jsonschema:"Start of time range (RFC3339 or relative like -24h). Default: -24h"`
    EndTime   string `json:"end_time,omitempty" jsonschema:"End of time range (RFC3339 or relative like now). Default: now"`
}

type DependencyLink struct {
    Caller    string `json:"caller" jsonschema:"Calling service name"`
    Callee    string `json:"callee" jsonschema:"Called service name"`
    CallCount uint64 `json:"call_count" jsonschema:"Number of calls from caller to callee in the time window"`
}

Two details that matter more than they look.

Accept relative time. -24h and now are what an LLM naturally produces when a user says "the last day." Forcing RFC3339 only means the model has to compute timestamps, which it does badly, and now you own a class of bugs where the agent silently queries the wrong window.

Sort the output. The handler sorts by caller then callee:

slices.SortFunc(links, func(a, b types.DependencyLink) int {
    if c := cmp.Compare(a.Caller, b.Caller); c != 0 {
        return c
    }
    return cmp.Compare(a.Callee, b.Callee)
})

Deterministic output is not cosmetic when the consumer is a language model. Map iteration order in Go is randomized, so an unsorted response gives a different token sequence for identical data on every call. That defeats prompt caching and makes agent behavior non-reproducible when you are trying to debug it. I hit the same issue in PR #8339, where search_traces was computing a set of service names to derive service_count and then throwing the names away. Surfacing them (sorted) removed an entire round of follow-up tool calls.

Testing that it actually speaks MCP

The existing tests posted hand-rolled JSON-RPC payloads over HTTP. That verifies your handlers. It does not verify that you implemented the protocol, because you are also the one writing the request.

PR #8230 added integration tests that drive the server through the real MCP Go SDK client (mcp.NewClient with StreamableClientTransport) against mock storage: protocol negotiation, tools/list discovery, typed invocation of every tool, concurrent session independence, and error paths.

The most valuable test is the one that walks the intended workflow end to end: services, then search, then topology, then critical path, then details. It is the only test that would fail if progressive disclosure quietly broke.

Two things I would tell anyone writing these:

Assert on the specific error, not just that one occurred. A test that only checks err != nil passes when your transport dies for unrelated reasons. Review caught exactly this in my missing-required-field test, which would have gone green on a session timeout.

Empty results must serialize as [], not null. A null where an agent expects a list is a parse error or, worse, a hallucinated recovery. It is worth a dedicated test.

Later, PR #8383 added end-to-end tracing tests: create a root span, format it as a W3C traceparent, send it through CallToolParams.Meta, and assert the middleware span has a matching TraceID and parent SpanID. That verifies trace context actually propagates from an MCP client into the server's own spans, which is what makes agent tool calls debuggable in production.

What I would take to your own MCP server

Wrap your service layer, not your storage. You inherit auth, tenancy, and semantics rather than reimplementing them.
Tier your tools by output size and say so in the descriptions. Context window is the scarce resource.
Put guidance in schemas, not system prompts. The description is read at the decision point.
Make truncation self-evident. Return a total alongside a possibly-truncated list.
Sort everything. Deterministic output is a correctness property when the consumer is a model.
Accept the time formats an LLM actually emits.
Test through a real SDK client, and test the intended workflow as a single path.

The code is all public, in cmd/jaeger/internal/extension/jaegerquery/internal/mcptools/. It has moved since I started (it was its own jaegermcp extension before being folded into the query extension), which is its own small lesson about contributing to an active CNCF project: write the tests well and the code survives the refactor.

Additionally,
I later benchmarked this question properly: jaeger-mcp-bench, 72 randomized trials across Claude and Gemini.

I do this work for hire.

I make LLM agents safe and observable when they touch production infrastructure. Two things I sell, both fixed-scope:

MCP server for your Go service: tool design, schemas models actually call correctly, OAuth, tracing, and integration tests that prove it works. ~2 weeks, from $3,000.
Agent tool-call audit: what your agents can actually reach, where the write paths are, what is untraced, and the blast radius if a model gets it wrong. Written report in 7 business days.

roshansingh.systems/#hire