95 Seconds to Root Cause: Claude Code Meets the SigNoz MCP

Connecting the SigNoz MCP server to Claude Code, a real latency hunt, and the traps that make the AI confidently wrong.

A service in my stack got slow: p99 latency jumped from ~120 ms to ~2 seconds after a deploy, and nothing was in the logs because nothing had actually failed. The old way to chase this is a half hour of clicking through traces, one span at a time, trying to hold a hypothesis in your head while the tabs pile up. I've done that afternoon. It's the kind of debugging that leaves you more tired than informed.

Somewhere around the twentieth trace, squinting at durations and muttering "there has to be a faster way to do this part," I realized the slow bit wasn't the thinking it was the grunt work of slicing traces a hundred different ways to find the one cut that mattered. So the next time felt different by design: instead of clicking, I typed one paragraph into Claude Code, and ninety seconds later it told me the exact culprit one payment provider's authorization path, pinned at a flat 2.0 seconds, starting at a specific minute. It even guessed the mechanism. I did the tedious version once so I could wire up the shortcut; what follows is that shortcut, plus every sharp edge I hit setting it up, so you don't have to experiment your way there.

The thing that makes it possible is something most people have never turned on: SigNoz ships an MCP server that lets an AI assistant query your traces, metrics, logs, alerts, and dashboards in natural language. It shipped in May 2026 and plugs into Claude Code in about two commands. This is the setup guide, a real worked example, and the mistakes that will bite you, my learnings, handed over so your first run goes better than mine did.

Here is the one lesson I want you to leave with, so I don't bury it: in the same breath, the copilot handed me a rock-solid root cause and a confident guess at the mechanism and one was right while the other was wrong. Knowing which to trust is the entire skill, and it comes down to two habits I'll show you: scope the time window, and verify the last mile yourself.

Everything here is self-hosted and tested: macOS, SigNoz v0.133.0, signoz-mcp-server v0.8.0.

What the MCP server actually is

MCP (Model Context Protocol) is an open standard for connecting AI assistants to external tools and data. SigNoz's server exposes 41 tools to the assistant: things like signoz_list_services, signoz_aggregate_traces, signoz_get_field_values, signoz_query_metrics, signoz_list_alerts, plus alert and dashboard management. You ask a question in plain English; Claude picks the tools and arguments and runs the queries. You never leave the terminal. That's the whole pitch the querying that used to be yours to do by hand becomes something you delegate.

Part 1: setup

1. Get the binary

Three options: a GitHub release, go install, or Docker. I grabbed the release for my platform (swap darwin_arm64 for darwin_amd64 or linux_amd64):

curl -L https://github.com/SigNoz/signoz-mcp-server/releases/latest/download/signoz-mcp-server_darwin_arm64.tar.gz | tar xz

Sanity check that it runs:

./signoz-mcp-server_darwin_arm64/bin/signoz-mcp-server --help
# Configuration validation failed: SIGNOZ_API_KEY is required for stdio mode

Good. It runs; it just needs a key.

2. Make an API key, and give it a role

In SigNoz: Settings → Service Accounts, create one, open its Keys tab, Add Key, and copy the value. Only admins can create these.

The first trap bit me immediately, before I'd even asked a real question: my brand-new key returned a 403 on the very first query.

SigNoz API error: unexpected status 403: only viewers/editors/admins can access this resource

A service account with no role can authenticate but can't read anything. Assign it a role. If you only want Claude to read telemetry, use Viewer (more on why in the traps).

Settings → Service Accounts. The key needs a role or every query 403s.

3. Add it to Claude Code

One command. Stdio mode means Claude launches the binary itself and passes the environment:

claude mcp add --scope user signoz "/full/path/to/signoz-mcp-server" \
  -e SIGNOZ_URL="http://localhost:8080" \
  -e SIGNOZ_API_KEY="<your-api-key>" \
  -e LOG_LEVEL=info

Confirm it connected:

claude mcp list
# signoz: /.../signoz-mcp-server  - ✔ Connected

One gotcha that cost me a few confused minutes: if you run claude mcp add from inside a live Claude session, the tools do not appear in that session. Start a fresh claude session and they load. Ask it to call signoz_list_services to confirm; you should get your services back with call rate, error rate, and p99.

Part 2: a real incident, investigated

Setup working is one thing; I wanted to know if it could actually carry its weight on the kind of incident that ruins an afternoon. So I built one on purpose. I have a small instrumented service, payments-api, with a /pay endpoint that calls a downstream /authorize ("the bank"). Each span carries custom attributes: payment.provider, payment.amount_cents, order.id.

Then I shipped a bad version the digital equivalent of the deploy that started my bad afternoon. A new risk-check path routes one provider, acme-bank, through a slow branch that waits two seconds before returning. It still returns 200. Nothing errors. The other providers are untouched.

@app.get("/authorize")
def authorize():
    provider = request.args.get("provider", "unknown")
    trace.get_current_span().set_attribute("payment.provider", provider)
    if BAD_DEPLOY and provider == "acme-bank":   # the bad deploy
        time.sleep(2.0)                            # slow new path, no error raised
    else:
        time.sleep(random.uniform(0.02, 0.12))
    return jsonify(authorized=True)

This is the nasty kind of incident, the one with nothing to grep for: p99 climbs, no errors, no log lines. You have to read the traces to explain it which, done by hand, is exactly the slog I was trying to escape. So I opened a fresh claude session and asked, in one paragraph:

Our payments-api got slow after a deploy a few minutes ago. p99 latency spiked but there are no errors. Investigate the last 15 minutes with the SigNoz MCP tools: confirm the regression and when it started, which operation is slow, and the most specific root cause from the span attributes. Give me an incident report with a fix.

It ran seven SigNoz queries in about 95 seconds (roughly 65 cents): p99 over time, p99 by operation, then p99 grouped by payment.provider the same progression I'd have clicked through myself, just without me clicking. Its report, lightly trimmed:

When it started: p99 in 60-second buckets held at ~125 to 132 ms through 17:54 UTC, then jumped to 2088 ms at 17:55:00 UTC, a 15.8x step with no ramp. That shape is a deploy, not organic load.

Which operation: p99 by span name over the window: GET /pay 2037 ms, charge.handler 2035 ms, GET /authorize 2015 ms, all one call path. GET / stayed at 1.18 ms.

Root cause: breaking the spike window down by payment.provider: acme-bank /authorize = 2040 ms, mastercard = 125 ms, visa = 122 ms. payment.provider = acme-bank is the entire regression. It's ~31% of traffic sitting at ~2s, which alone drags the service p99 to ~2s. The flat 2.0s floor with zero errors is a configured constant, not a saturated dependency.

Fix: roll back the 17:55 deploy and diff the acme-bank client config for a newly introduced 2000 ms value, most likely a retry backoff or a timeout on the wrong code path.

That is correct, from the trace attributes, in under two minutes. The real bug is literally a two-second sleep on the acme-bank branch, and it found the provider, the path, the onset minute, and the "it's a constant, not saturation" character without me pointing at any of it. The half-hour afternoon, compressed to the length of a coffee.

Claude calling signoz_aggregate_traces, grouping p99 by payment.provider, and landing on acme-bank.

The same regression in the UI: p99 steps from ~120 ms to ~2 s, error rate flat at zero.

A trace for provider=acme-bank: the two seconds sit entirely in the /authorize span, it returns 200, and the span attribute payment.provider: "acme-bank" is right there in the panel. That attribute is what lets the copilot name the culprit.

Part 3: the traps (the part that actually matters)

The run above worked because I'd already paid for a few lessons the hard way. Here's what goes wrong when you skip them so you can borrow the lessons instead of buying them.

Scope the time window, always

I asked about "the last 15 minutes." That is why Claude pulled 60-second buckets and could name the onset to the minute (17:55:00 UTC). Left unbounded, the server defaults to a wide window (hours), and a sharp recent regression gets averaged into the baseline or mislocated against older data. The single biggest quality lever on these investigations is telling it when to look. Give it a window, or a "since HH:MM," every time.

It infers the mechanism from the shape. Verify the last mile

Notice the fix said "a retry backoff or a timeout." My actual bug was neither it was a hard sleep but the shape it reasoned from (a flat 2.0s floor, zero errors, one provider) was exactly right, and it correctly called it a "configured constant." It even flagged, on its own, that it hadn't opened a single trace to confirm the mechanism. This is the whole trust question in miniature: treat the copilot's root-cause dimension (acme-bank, the authorize path, 17:55 UTC) as solid evidence, and its proposed mechanism as a strong lead to confirm, not a fact. One signoz_get_trace_details call would have settled it.

The tools only load in a fresh session

Add the server, then start a new claude. If you add it mid-session and the tools "aren't there," this is why. It is not broken it just cost me a few minutes of doubting my own setup, and I'd rather it not cost you any.

Mind what the key can do

The server includes write tools: signoz_create_alert, signoz_delete_dashboard, signoz_delete_alert. If your key has an admin or editor role, Claude can call them. For everyday debugging, a Viewer key keeps the copilot strictly read-only and removes any chance of it "helpfully" changing your monitoring. Use a higher role only when you actually want it managing alerts and dashboards.

It costs real calls and time

A genuine investigation is not one query. Mine was seven tool calls, 95 seconds, about 65 cents. A broader, vaguer question balloons that fast. Narrow the question and the window, both for a better answer and a shorter loop. One more thing you'll notice in the transcript: SigNoz timestamps come back as Unix milliseconds, and Claude will shell out to convert them, so don't be surprised by a stray python3 date command mid-investigation.

What I'd tell you before you start

If we were pairing and you were about to do this for the first time, here's everything I'd say over your shoulder:

• The MCP server is a separate binary. Download it; it is not part of the SigNoz stack.
• A key with no role fails with a 403 on the first query. Assign Viewer for read-only.
• Add the server, then start a fresh claude session so the tools load.
• Put a time window in every prompt. It is the biggest lever on answer quality.
• Trust the dimension it isolates; verify the mechanism it guesses.
• Instrument custom attributes (like payment.provider). They are what let the copilot say "acme-bank" instead of "latency is up."

Conclusion

The SigNoz MCP server turns Claude Code into a fast, capable first responder. On a silent latency regression the kind with no error to grep for it read the traces, grouped by the right attribute, and handed me the provider, the path, the minute it started, and a plausible fix in the time it takes to make coffee. It is not magic: a vague, unbounded question gets a confident, half-right answer. But a scoped question against well-instrumented services gets you most of the way to root cause before you've opened a single tab. I did the slow version once so I'd never have to again and now you don't have to do it even the first time.

Docs: signoz.io/docs/ai/signoz-mcp-server · Repo: github.com/SigNoz/signoz-mcp-server

Comments

[Mads Hansen]

The provider split is strong evidence, but I would require one counterfactual before calling it root cause: show acme-bank latency and sample count before and after 17:55, alongside the other providers, under the same query. A post-deploy p99 split alone can be distorted by low counts or a traffic-mix change. The incident report should preserve the exact time range, filters, aggregation, group-by, sample counts, deployment marker, and representative trace IDs, with observations separated from inferred mechanism. That makes the 95-second diagnosis reproducible and lets a human verify the evidence without replaying the whole investigation.