DEV Community: Roshan Singh

Two ceilings: taking a Go DNS server from 500 to 9,500 QPS

Roshan Singh — Wed, 29 Jul 2026 06:13:42 +0000

I run HydraDNS, an open-source DNS security gateway in Go. Last month I sat down to find out what one box could actually handle before I put it on anyone else's network. The plan had a rule I'd written for myself: every number we discover becomes either a sales claim or a fix ticket. No number, no claim.

I expected to find one bottleneck. I found two, stacked on top of each other, and a third thing I wasn't looking for: a data structure in our own documentation that had never existed in the code.

Everything below was measured on a 22-core dev machine with load generated inside the container, using dnspyre, so docker-proxy and host networking stay out of the numbers. It's not appliance hardware and I'm not making appliance claims. The shapes are what matter.

The first ceiling: ~500 QPS, and it didn't care what I threw at it

The first redline run capped at roughly 500 queries per second. Fine, servers have limits. What made it interesting was that the cap didn't move. Blocked queries: ~500. Cached queries that never touch upstream: ~500. Two code paths that do completely different work, hitting the same wall, with the CPU sitting under 30% of 22 cores.

That signature is worth memorizing. When two very different paths hit the same ceiling and the CPU is bored, the bottleneck isn't in either path. It's in something they share, or something upstream of both.

Ours was in the blocklist check. IsBlocked ran a SQL COUNT against a 92k-row blocklist_entries table on every query. Not just candidate blocks, every query, because the check sits in front of the cache, so even cache hits paid for it. And all of those reads were serialized through a single SQLite connection, MaxOpenConns=1, which was also absorbing the async write traffic from query logging.

The engine's self-measured latency under load: p50 of 50ms, p99 of 5000ms. Five full seconds at the tail, for DNS, which is supposed to be the fast part of the internet.

The part where I found out our docs were lying

Here's the uncomfortable bit. Our feature sheet, and two other internal docs, said the blocklist was backed by a Bloom filter. Sub-millisecond membership checks, the whole pitch. Neither thing was true.

There was no Bloom filter. There never had been one. The policy engine has one, a real one, with tests. The blocklist never did. Somewhere along the way "we should use a Bloom filter" became "we use a Bloom filter" in a document, and then the claim got copied into another document, and after it existed in three places it read like an established fact. Three docs, zero source files.

Nobody lied on purpose. A plausible sentence got written once and nothing ever checked it against the code. I only caught it because the stress test forced me to look at what the hot path actually did, and what it actually did was a table scan's worth of SQL per packet.

So the fix wasn't "tune the Bloom filter." The fix was to build the in-memory layer the docs had been imagining. MemoryChecker: an atomic in-memory domain set, loaded at startup, swapped wholesale on the existing 6-hour blocklist refresh. The parent-domain walk (block example.com, and ads.example.com matches too) reproduces exactly the candidate set the SQL version checked, so behavior didn't change, just the cost.

Throughput went from ~500 to ~9,500 QPS. Nineteen times, from moving one membership check off the database. Not a clever data structure. A map, held in memory, reloaded every six hours.

The second ceiling was hiding behind the first

With reads fixed, I re-ran the redline, and the box got fast and then started eating memory. RSS climbed from 98MiB to a bit over 2GB as offered load went from 500 to 10,000 QPS.

It drained back to normal when the load stopped, so it wasn't a leak, it was a backlog. But a 2GB transient spike is academic only until you remember the target hardware has 2 to 4GB of RAM total. On the appliance this is an OOM kill in the middle of a traffic burst, which is the exact moment you don't want your DNS to die.

The cause: query logging spawned a goroutine per query, and each one did an INSERT plus a stats UPDATE through that same single SQLite connection. At 500 QPS the queue drained fine and nobody noticed. At 10,000 QPS, goroutines piled up faster than one connection could ever clear them. The first bottleneck had been rationing the second one. Lift the read ceiling and the write bomb goes off.

"Async logging" is only safe if it's bounded. Unbounded goroutine-per-event isn't async, it's a memory spike with extra steps, and it just moves the failure from slow to OOM.

The replacement is querylog_writer.go: one writer goroutine, a bounded channel of 4,096 entries, batches flushed at 256 entries or 500ms, bulk INSERT plus a single aggregated stats UPDATE per batch. Enqueue is non-blocking. If the buffer is full, the entry is dropped and the drop is counted, because losing a log line is an acceptable failure for a DNS server and stalling resolution is not.

After: RSS flat at ~85MiB from 500 to 10,000 QPS. Engine latency p50 5ms, p99 20ms. CPU at 10k QPS dropped from 56% to 14%, because it turns out scheduling tens of thousands of goroutines was itself a workload.

The numbers, before and after

Throughput ceiling:   ~500  -> ~9,500 QPS   (19x)
Engine p99:           5000ms -> 20ms
RSS at 10k QPS:       2,063MiB -> ~85MiB
CPU at 10k QPS:       56% -> 14%

Then a 3-minute soak to confirm the shape holds: 248,166 queries at 1,379 QPS, zero engine errors, RSS a bounded sawtooth (119 to 139 to 114MiB as GC does its thing) instead of the old monotonic climb.

The soak that almost passed on an idle box

One more confession, because it's the most useful lesson in here. My first soak attempt reported beautifully. Flat memory, no errors, perfect.

It had also sent zero queries. Deploying the retention change had recreated the container, which wiped the dnspyre binary inside it, and the docker exec failures were being swallowed by a 2>/dev/null I'd added earlier. The harness "ran," measured an idle server, and called it healthy. I caught it only because the sent-count looked suspiciously round: zero.

If your load generator's failure mode is silence, your benchmark's failure mode is a false pass. Now the harness asserts the sent count before it's allowed to report anything.

What I'd take to your codebase

Same ceiling on two different paths plus idle CPU means the bottleneck is shared or upstream. Stop optimizing the paths.
Fixing one bottleneck unmasks the next. The box's real limit is wherever you stop looking.
Bound every async path. A queue without a cap is an OOM with a delay on it.
Dropping data beats stalling the hot path, if you count the drops.
Verify advertised internals against the code. Our "Bloom filter" lived in three documents and zero files, and it survived because every reader checked it against another document instead of the source.

That last one is the one I keep thinking about. The expensive bug wasn't in the code. It was in the docs, quietly shaping what everyone believed the code did.

I do this work for hire, mostly making LLM agents safe and observable against production infrastructure, and occasionally chasing a database off a hot path. Scope and pricing are at roshansingh.systems/#hire, or write to inbox@roshansingh.systems.

Building Jaeger's MCP server: connecting LLMs to distributed traces

Roshan Singh — Tue, 28 Jul 2026 13:40:00 +0000

A distributed trace is one of the few places where a system tells you the truth about itself. It records what actually called what, in what order, and how long each hop took. It is also, for a human being at 2am, a wall of a thousand spans.

That combination makes tracing data an unusually good target for an LLM. The data is structured, causal, and too large to read by hand. Over the last few months I shipped seven merged PRs into CNCF Jaeger's MCP server, the component that lets an LLM query traces as a first-class tool rather than by scraping the UI.

This post is about how that server is built in Go, and about the one design lesson that took me three PRs and a maintainer's review to actually internalize.

What the server is

The Model Context Protocol (MCP) is a way to hand an LLM a typed set of tools with schemas, instead of an API and hope. Jaeger's implementation lives in the main Go binary as a package that wraps the existing QueryService:

// Package mcptools provides the Jaeger telemetry MCP tools as a reusable
// library. The tools wrap a *querysvc.QueryService.
package mcptools

func NewServer(telset telemetry.Settings, queryAPI *querysvc.QueryService, cfg Config) *mcp.Server {
    server := mcp.NewServer(
        &mcp.Implementation{
            Name:    cfg.ServerName,
            Version: cfg.ServerVersion,
        },
        &mcp.ServerOptions{
            Instructions: serverInstructions,
        },
    )
    registerTools(server, queryAPI, cfg)
    // ...
    server.AddReceivingMiddleware(mw...)
    return server
}

Three things are worth pulling out of that.

It takes a *querysvc.QueryService directly, not a storage handle. The MCP tools do not reach into Cassandra or Elasticsearch. They go through the same query layer the Jaeger UI uses, which means tenancy, storage backends, and query semantics are all inherited for free. This was a maintainer correction early on: the original ADR had the extension depending on jaegerstorage, and it was changed to depend on the query service instead. Wrapping the layer that already has the business logic is almost always the right call for an MCP server.

Instructions are embedded, not configured. serverInstructions is a //go:embed INSTRUCTIONS.md string handed to ServerOptions.Instructions, which the SDK returns during the MCP initialize handshake. The LLM client receives the server's usage guidance automatically. Nobody has to paste anything into a config file.

Middleware is the observability seam. The server registers receiving middleware for tracing and metrics. There is a pleasing recursion in a tracing backend emitting traces about the tool calls an agent makes against it, and it turns out to be genuinely useful for debugging agent behavior.

The transport is streamable HTTP, wrapped with tenancy extraction and otelhttp, and mounted on an existing mux rather than binding its own listener:

func WrapHTTP(server *mcp.Server, tenancyMgr *tenancy.Manager, telset telemetry.Settings) http.Handler {
    mcpHandler := mcp.NewStreamableHTTPHandler(
        func(*http.Request) *mcp.Server { return server },
        &mcp.StreamableHTTPOptions{
            JSONResponse:   false, // Use SSE for streamed events
            Stateless:      false, // Session state management
            SessionTimeout: mcpSessionTimeout,
        },
    )
    tenantHandler := tenancy.ExtractTenantHTTPHandler(tenancyMgr, mcpHandler)
    return otelhttp.NewHandler(
        tenantHandler,
        "jaeger_mcp",
        otelhttp.WithTracerProvider(telset.TracerProvider),
    )
}

One server instance is reused across sessions. With Stateless: false the SDK builds one ServerSession per MCP session and reuses it for that session's requests.

The tools, and why they are shaped that way

The server currently registers nine tools. The interesting part is not the list, it is that the list is deliberately tiered by cost:

get_services, get_span_names - cheap discovery
search_traces - lightweight summaries, no spans or attributes
get_trace_topology - trace structure as a flat depth-first list, no attributes
get_critical_path - the chain of spans that determined end-to-end duration
get_trace_errors, get_span_details - verbose, full OTLP data
get_service_dependencies - the service graph
read_skill - built-in analysis playbooks

This is progressive disclosure, and it exists for one reason: context window economy. A topology span is roughly 185 bytes. A full span detail is roughly 1KB. When I measured that during the instructions work, the ratio was about 6x. An agent that calls get_span_details on every span in a 500-span trace burns its entire context before it has formed a hypothesis. An agent that starts with topology, finds the critical path, and then fetches details for four spans, answers the question.

So the whole design problem becomes: how do you get the LLM to drill down instead of dumping?

My first answer was wrong.

The lesson: the schema is the interface, not the prompt

I opened PR #8253 to add an INSTRUCTIONS.md system prompt. My draft was thorough in the way that draft documentation is thorough: a tool reference table, investigation patterns, a constraints section, a system limits section. It explained the drill-down order carefully. It was, I thought, the fix.

The maintainer review pushed back on the size, and I trimmed it. Then came the comment that actually mattered:

we could improve some tool descriptions to be more explicit about the limits and truncation. For example [...] The ErrorCount could be renamed to TotalErrorCount and the Spans list could say (possibly truncated to MaxXyz size)

That reframed the problem. I had been writing a document about the tools, sitting outside them, that an agent may or may not have in context by the time it decides what to call. The information belonged in the tools, in the schema, at the exact point of decision.

INSTRUCTIONS.md shipped at fifteen lines. Here is essentially all of it:

Jaeger is a distributed tracing backend. A trace is a tree of spans representing a request or workflow [...] These tools support progressive disclosure to manage context density. While they can be called in any order based on available data, prefer starting with broad discovery (get_services or search_traces) or structural overviews (get_trace_topology) before requesting verbose OTLP details for specific spans.

Everything else moved into the schemas. The trace errors output went from a bare ErrorCount to this:

type GetTraceErrorsOutput struct {
    TraceID         string       `json:"trace_id" jsonschema:"Unique identifier for the trace"`
    TotalErrorCount int          `json:"total_error_count" jsonschema:"Total number of error spans in the trace (may exceed the size of the spans list due to per-request limits)"`
    Spans           []SpanDetail `json:"spans,omitempty" jsonschema:"Error span details (possibly truncated to server-configured limit)"`
}

Read that as an agent. total_error_count tells you there were 40 errors. The spans schema tells you the list you are holding may be truncated. The agent can now detect its own truncation by comparing the two numbers, without being told to in a prompt it might not be attending to.

I followed up in PR #8314 by rewriting all the tool descriptions on the same principle. Each one now states what the tool does, what distinguishes its output, and the runtime behavior the schema cannot express:

mcp.AddTool(s.mcpServer, &mcp.Tool{
    Name: "get_trace_topology",
    Description: "Get the structural overview of a trace as a flat, depth-first span list. " +
        "Each span includes a 'path' field encoding ancestry as slash-delimited span IDs " +
        "(e.g. 'rootID/parentID/spanID'). " +
        "Does NOT include attributes, events, or links.",
}, handlers.NewGetTraceTopologyHandler(s.queryAPI, s.config.MaxSpanDetailsPerRequest))

Note what is not there: no workflow ordering, no "call this before that." The input and output schemas already encode the dependencies. Sequencing instructions in a description are a smell that your schema is underspecified.

The generalizable rule: a tool description is read at the moment of the decision it governs. A system prompt is read once, at the start, and competes with everything since. Put the guidance where the decision happens.

Adding a tool, end to end

get_service_dependencies (PR #8403) is a clean example of the whole shape. QueryService.GetDependencies() was the only query method with no MCP tool, so an agent asking "what does payment-service call?" had to fetch traces and rebuild the graph by hand, even though Jaeger already computes and stores it.

The types come first, because in MCP the types are the API:

type GetDependenciesInput struct {
    StartTime string `json:"start_time,omitempty" jsonschema:"Start of time range (RFC3339 or relative like -24h). Default: -24h"`
    EndTime   string `json:"end_time,omitempty" jsonschema:"End of time range (RFC3339 or relative like now). Default: now"`
}

type DependencyLink struct {
    Caller    string `json:"caller" jsonschema:"Calling service name"`
    Callee    string `json:"callee" jsonschema:"Called service name"`
    CallCount uint64 `json:"call_count" jsonschema:"Number of calls from caller to callee in the time window"`
}

Two details that matter more than they look.

Accept relative time. -24h and now are what an LLM naturally produces when a user says "the last day." Forcing RFC3339 only means the model has to compute timestamps, which it does badly, and now you own a class of bugs where the agent silently queries the wrong window.

Sort the output. The handler sorts by caller then callee:

slices.SortFunc(links, func(a, b types.DependencyLink) int {
    if c := cmp.Compare(a.Caller, b.Caller); c != 0 {
        return c
    }
    return cmp.Compare(a.Callee, b.Callee)
})

Deterministic output is not cosmetic when the consumer is a language model. Map iteration order in Go is randomized, so an unsorted response gives a different token sequence for identical data on every call. That defeats prompt caching and makes agent behavior non-reproducible when you are trying to debug it. I hit the same issue in PR #8339, where search_traces was computing a set of service names to derive service_count and then throwing the names away. Surfacing them (sorted) removed an entire round of follow-up tool calls.

Testing that it actually speaks MCP

The existing tests posted hand-rolled JSON-RPC payloads over HTTP. That verifies your handlers. It does not verify that you implemented the protocol, because you are also the one writing the request.

PR #8230 added integration tests that drive the server through the real MCP Go SDK client (mcp.NewClient with StreamableClientTransport) against mock storage: protocol negotiation, tools/list discovery, typed invocation of every tool, concurrent session independence, and error paths.

The most valuable test is the one that walks the intended workflow end to end: services, then search, then topology, then critical path, then details. It is the only test that would fail if progressive disclosure quietly broke.

Two things I would tell anyone writing these:

Assert on the specific error, not just that one occurred. A test that only checks err != nil passes when your transport dies for unrelated reasons. Review caught exactly this in my missing-required-field test, which would have gone green on a session timeout.

Empty results must serialize as [], not null. A null where an agent expects a list is a parse error or, worse, a hallucinated recovery. It is worth a dedicated test.

Later, PR #8383 added end-to-end tracing tests: create a root span, format it as a W3C traceparent, send it through CallToolParams.Meta, and assert the middleware span has a matching TraceID and parent SpanID. That verifies trace context actually propagates from an MCP client into the server's own spans, which is what makes agent tool calls debuggable in production.

What I would take to your own MCP server

Wrap your service layer, not your storage. You inherit auth, tenancy, and semantics rather than reimplementing them.
Tier your tools by output size and say so in the descriptions. Context window is the scarce resource.
Put guidance in schemas, not system prompts. The description is read at the decision point.
Make truncation self-evident. Return a total alongside a possibly-truncated list.
Sort everything. Deterministic output is a correctness property when the consumer is a model.
Accept the time formats an LLM actually emits.
Test through a real SDK client, and test the intended workflow as a single path.

The code is all public, in cmd/jaeger/internal/extension/jaegerquery/internal/mcptools/. It has moved since I started (it was its own jaegermcp extension before being folded into the query extension), which is its own small lesson about contributing to an active CNCF project: write the tests well and the code survives the refactor.

I do this work for hire.

I make LLM agents safe and observable when they touch production infrastructure. Two things I sell, both fixed-scope:

MCP server for your Go service: tool design, schemas models actually call correctly, OAuth, tracing, and integration tests that prove it works. ~2 weeks, from $3,000.
Agent tool-call audit: what your agents can actually reach, where the write paths are, what is untraced, and the blast radius if a model gets it wrong. Written report in 7 business days.

roshansingh.systems/#hire