Building Jaeger's MCP server: connecting LLMs to distributed traces

A distributed trace is one of the few places where a system tells you the truth about itself. It records what actually called what, in what order, and how long each hop took. It is also, for a human being at 2am, a wall of a thousand spans.

That combination makes tracing data an unusually good target for an LLM. The data is structured, causal, and too large to read by hand. Over the last few months I shipped seven merged PRs into CNCF Jaeger's MCP server, the component that lets an LLM query traces as a first-class tool rather than by scraping the UI.

This post is about how that server is built in Go, and about the one design lesson that took me three PRs and a maintainer's review to actually internalize.

What the server is

The Model Context Protocol (MCP) is a way to hand an LLM a typed set of tools with schemas, instead of an API and hope. Jaeger's implementation lives in the main Go binary as a package that wraps the existing QueryService:

// Package mcptools provides the Jaeger telemetry MCP tools as a reusable
// library. The tools wrap a *querysvc.QueryService.
package mcptools

func NewServer(telset telemetry.Settings, queryAPI *querysvc.QueryService, cfg Config) *mcp.Server {
    server := mcp.NewServer(
        &mcp.Implementation{
            Name:    cfg.ServerName,
            Version: cfg.ServerVersion,
        },
        &mcp.ServerOptions{
            Instructions: serverInstructions,
        },
    )
    registerTools(server, queryAPI, cfg)
    // ...
    server.AddReceivingMiddleware(mw...)
    return server
}

Three things are worth pulling out of that.

It takes a *querysvc.QueryService directly, not a storage handle. The MCP tools do not reach into Cassandra or Elasticsearch. They go through the same query layer the Jaeger UI uses, which means tenancy, storage backends, and query semantics are all inherited for free. This was a maintainer correction early on: the original ADR had the extension depending on jaegerstorage, and it was changed to depend on the query service instead. Wrapping the layer that already has the business logic is almost always the right call for an MCP server.

Instructions are embedded, not configured. serverInstructions is a //go:embed INSTRUCTIONS.md string handed to ServerOptions.Instructions, which the SDK returns during the MCP initialize handshake. The LLM client receives the server's usage guidance automatically. Nobody has to paste anything into a config file.

Middleware is the observability seam. The server registers receiving middleware for tracing and metrics. There is a pleasing recursion in a tracing backend emitting traces about the tool calls an agent makes against it, and it turns out to be genuinely useful for debugging agent behavior.

The transport is streamable HTTP, wrapped with tenancy extraction and otelhttp, and mounted on an existing mux rather than binding its own listener:

func WrapHTTP(server *mcp.Server, tenancyMgr *tenancy.Manager, telset telemetry.Settings) http.Handler {
    mcpHandler := mcp.NewStreamableHTTPHandler(
        func(*http.Request) *mcp.Server { return server },
        &mcp.StreamableHTTPOptions{
            JSONResponse:   false, // Use SSE for streamed events
            Stateless:      false, // Session state management
            SessionTimeout: mcpSessionTimeout,
        },
    )
    tenantHandler := tenancy.ExtractTenantHTTPHandler(tenancyMgr, mcpHandler)
    return otelhttp.NewHandler(
        tenantHandler,
        "jaeger_mcp",
        otelhttp.WithTracerProvider(telset.TracerProvider),
    )
}

One server instance is reused across sessions. With Stateless: false the SDK builds one ServerSession per MCP session and reuses it for that session's requests.

The tools, and why they are shaped that way

The server currently registers nine tools. The interesting part is not the list, it is that the list is deliberately tiered by cost:

• get_services, get_span_names - cheap discovery
• search_traces - lightweight summaries, no spans or attributes
• get_trace_topology - trace structure as a flat depth-first list, no attributes
• get_critical_path - the chain of spans that determined end-to-end duration
• get_trace_errors, get_span_details - verbose, full OTLP data
• get_service_dependencies - the service graph
• read_skill - built-in analysis playbooks

This is progressive disclosure, and it exists for one reason: context window economy. A topology span is roughly 185 bytes. A full span detail is roughly 1KB. When I measured that during the instructions work, the ratio was about 6x. An agent that calls get_span_details on every span in a 500-span trace burns its entire context before it has formed a hypothesis. An agent that starts with topology, finds the critical path, and then fetches details for four spans, answers the question.

So the whole design problem becomes: how do you get the LLM to drill down instead of dumping?

My first answer was wrong.

The lesson: the schema is the interface, not the prompt

I opened PR #8253 to add an INSTRUCTIONS.md system prompt. My draft was thorough in the way that draft documentation is thorough: a tool reference table, investigation patterns, a constraints section, a system limits section. It explained the drill-down order carefully. It was, I thought, the fix.

The maintainer review pushed back on the size, and I trimmed it. Then came the comment that actually mattered:

we could improve some tool descriptions to be more explicit about the limits and truncation. For example [...] The ErrorCount could be renamed to TotalErrorCount and the Spans list could say (possibly truncated to MaxXyz size)

That reframed the problem. I had been writing a document about the tools, sitting outside them, that an agent may or may not have in context by the time it decides what to call. The information belonged in the tools, in the schema, at the exact point of decision.

INSTRUCTIONS.md shipped at fifteen lines. Here is essentially all of it:

Jaeger is a distributed tracing backend. A trace is a tree of spans representing a request or workflow [...] These tools support progressive disclosure to manage context density. While they can be called in any order based on available data, prefer starting with broad discovery (get_services or search_traces) or structural overviews (get_trace_topology) before requesting verbose OTLP details for specific spans.

Everything else moved into the schemas. The trace errors output went from a bare ErrorCount to this:

type GetTraceErrorsOutput struct {
    TraceID         string       `json:"trace_id" jsonschema:"Unique identifier for the trace"`
    TotalErrorCount int          `json:"total_error_count" jsonschema:"Total number of error spans in the trace (may exceed the size of the spans list due to per-request limits)"`
    Spans           []SpanDetail `json:"spans,omitempty" jsonschema:"Error span details (possibly truncated to server-configured limit)"`
}

Read that as an agent. total_error_count tells you there were 40 errors. The spans schema tells you the list you are holding may be truncated. The agent can now detect its own truncation by comparing the two numbers, without being told to in a prompt it might not be attending to.

I followed up in PR #8314 by rewriting all the tool descriptions on the same principle. Each one now states what the tool does, what distinguishes its output, and the runtime behavior the schema cannot express:

mcp.AddTool(s.mcpServer, &mcp.Tool{
    Name: "get_trace_topology",
    Description: "Get the structural overview of a trace as a flat, depth-first span list. " +
        "Each span includes a 'path' field encoding ancestry as slash-delimited span IDs " +
        "(e.g. 'rootID/parentID/spanID'). " +
        "Does NOT include attributes, events, or links.",
}, handlers.NewGetTraceTopologyHandler(s.queryAPI, s.config.MaxSpanDetailsPerRequest))

Note what is not there: no workflow ordering, no "call this before that." The input and output schemas already encode the dependencies. Sequencing instructions in a description are a smell that your schema is underspecified.

The generalizable rule: a tool description is read at the moment of the decision it governs. A system prompt is read once, at the start, and competes with everything since. Put the guidance where the decision happens.

Adding a tool, end to end

get_service_dependencies (PR #8403) is a clean example of the whole shape. QueryService.GetDependencies() was the only query method with no MCP tool, so an agent asking "what does payment-service call?" had to fetch traces and rebuild the graph by hand, even though Jaeger already computes and stores it.

The types come first, because in MCP the types are the API:

type GetDependenciesInput struct {
    StartTime string `json:"start_time,omitempty" jsonschema:"Start of time range (RFC3339 or relative like -24h). Default: -24h"`
    EndTime   string `json:"end_time,omitempty" jsonschema:"End of time range (RFC3339 or relative like now). Default: now"`
}

type DependencyLink struct {
    Caller    string `json:"caller" jsonschema:"Calling service name"`
    Callee    string `json:"callee" jsonschema:"Called service name"`
    CallCount uint64 `json:"call_count" jsonschema:"Number of calls from caller to callee in the time window"`
}

Two details that matter more than they look.

Accept relative time. -24h and now are what an LLM naturally produces when a user says "the last day." Forcing RFC3339 only means the model has to compute timestamps, which it does badly, and now you own a class of bugs where the agent silently queries the wrong window.

Sort the output. The handler sorts by caller then callee:

slices.SortFunc(links, func(a, b types.DependencyLink) int {
    if c := cmp.Compare(a.Caller, b.Caller); c != 0 {
        return c
    }
    return cmp.Compare(a.Callee, b.Callee)
})

Deterministic output is not cosmetic when the consumer is a language model. Map iteration order in Go is randomized, so an unsorted response gives a different token sequence for identical data on every call. That defeats prompt caching and makes agent behavior non-reproducible when you are trying to debug it. I hit the same issue in PR #8339, where search_traces was computing a set of service names to derive service_count and then throwing the names away. Surfacing them (sorted) removed an entire round of follow-up tool calls.

Testing that it actually speaks MCP

The existing tests posted hand-rolled JSON-RPC payloads over HTTP. That verifies your handlers. It does not verify that you implemented the protocol, because you are also the one writing the request.

PR #8230 added integration tests that drive the server through the real MCP Go SDK client (mcp.NewClient with StreamableClientTransport) against mock storage: protocol negotiation, tools/list discovery, typed invocation of every tool, concurrent session independence, and error paths.

The most valuable test is the one that walks the intended workflow end to end: services, then search, then topology, then critical path, then details. It is the only test that would fail if progressive disclosure quietly broke.

Two things I would tell anyone writing these:

Assert on the specific error, not just that one occurred. A test that only checks err != nil passes when your transport dies for unrelated reasons. Review caught exactly this in my missing-required-field test, which would have gone green on a session timeout.

Empty results must serialize as [], not null. A null where an agent expects a list is a parse error or, worse, a hallucinated recovery. It is worth a dedicated test.

Later, PR #8383 added end-to-end tracing tests: create a root span, format it as a W3C traceparent, send it through CallToolParams.Meta, and assert the middleware span has a matching TraceID and parent SpanID. That verifies trace context actually propagates from an MCP client into the server's own spans, which is what makes agent tool calls debuggable in production.

What I would take to your own MCP server

1. Wrap your service layer, not your storage. You inherit auth, tenancy, and semantics rather than reimplementing them.
2. Tier your tools by output size and say so in the descriptions. Context window is the scarce resource.
3. Put guidance in schemas, not system prompts. The description is read at the decision point.
4. Make truncation self-evident. Return a total alongside a possibly-truncated list.
5. Sort everything. Deterministic output is a correctness property when the consumer is a model.
6. Accept the time formats an LLM actually emits.
7. Test through a real SDK client, and test the intended workflow as a single path.

The code is all public, in cmd/jaeger/internal/extension/jaegerquery/internal/mcptools/. It has moved since I started (it was its own jaegermcp extension before being folded into the query extension), which is its own small lesson about contributing to an active CNCF project: write the tests well and the code survives the refactor.

---

I do this work for hire.

I make LLM agents safe and observable when they touch production infrastructure. Two things I sell, both fixed-scope:

• MCP server for your Go service: tool design, schemas models actually call correctly, OAuth, tracing, and integration tests that prove it works. ~2 weeks, from $3,000.
• Agent tool-call audit: what your agents can actually reach, where the write paths are, what is untraced, and the blast radius if a model gets it wrong. Written report in 7 business days.

roshansingh.systems/#hire