DEV Community

Ganesh J
Ganesh J

Posted on

I gave an AI SRE copilot write access to SigNoz — here's what the MCP server actually returns

#ai #webdev #opentelemetry #observability

AI agents fail in ways classic monitoring misses. A tool call silently retries. A prompt balloons your token bill 4×. A judge step times out. An agent quietly loops. Your observability platform has all of it — but staring at a trace waterfall at 2am isn't the same as knowing what broke.

So for WeMakeDevs' Agents of SigNoz, I built Cerberus: an AI SRE copilot that reads your agent's telemetry back out of SigNoz, explains the incident in plain English with trace citations, and then writes the alert rule that would have caught it — all through SigNoz's own MCP server.

Live demo: https://cerberus-sre.vercel.app
Code: https://github.com/CodeMuscle/cerberus

screenshot: dashboard.png*

The idea: emit and consume

Most observability projects only emit telemetry — they push spans and stop. Cerberus does the agent-native thing: it emits, then consumes its own agent's traces back and acts on them. Three heads, three jobs:

  • Observe — every agent run lands in SigNoz as OpenTelemetry traces, with token usage, cost, latency and errors per step.
  • Explain — a copilot reads the ranked incidents and answers "what just went wrong?" grounded only in the facts, citing the exact trace_id.
  • Prevent — one click writes the SigNoz alert rule for the incident's root cause.

Why MCP, not the REST API

The interesting decision was the read path. SigNoz ships an MCP server — the same tool surface an AI client gets: signoz_execute_builder_query, signoz_search_traces, signoz_create_alert, and ~40 more. Instead of scraping a REST endpoint, Cerberus speaks MCP. That means the exact interface an LLM would use is the interface the product is built on.

Deploying it is one command, because SigNoz's Foundry installer brings up the whole stack plus the MCP server from a single casting file:

curl -fsSL https://signoz.io/foundry.sh | bash
foundryctl cast -f casting.yaml     # SigNoz on :8080, MCP on :8000
Enter fullscreen mode Exit fullscreen mode

casting.yaml and its lockfile are committed, so the deployment is reproducible — anyone can re-run it.

The part nobody tells you: the payload is not the docs

Here's what actually cost me a day, and what I'd want another builder to know.

1. signoz_search_traces returns a fixed column projection — with no gen_ai.* attributes. The obvious tool for "give me the spans" hands back a fixed set of columns. My token and cost data — gen_ai.usage.input_tokens, gen_ai.usage.cost_usd — came back empty every time. The whole product is about token and cost spikes, and they were zero.

The fix was to drop to the Query Builder v5 tool and select those columns explicitly:

{"name": "gen_ai.usage.input_tokens", "fieldDataType": "number",
 "signal": "traces", "fieldContext": "tag"}
Enter fullscreen mode Exit fullscreen mode

signoz_execute_builder_query lets you name exactly the attributes you want. Once I selected the gen_ai.* tags by hand, the real numbers showed up.

2. The response carries a human-readable note after the JSON. The payload looks like this:

{ ...valid JSON... }
note: returned 3 rows (limit 3) — more results likely exist (hasMore=true).
Enter fullscreen mode Exit fullscreen mode

A naive json.loads() throws on the trailing text — and if you catch that broadly, you silently get zero spans and no error. The fix is to decode a prefix (JSONDecoder().raw_decode) instead of parsing the whole string.

3. Rows nest one level down, and an empty window returns null. Each row wraps its fields under a data sub-object alongside a timestamp, and an empty time window comes back as "rows": null rather than [] — which will happily crash a flat-map if you assume a list. One or [] guard fixes it.

None of these are in the docs. All three are captured as tests against real captured payloads, so they don't regress.

Grounded, not hallucinated

The copilot doesn't get raw traces dumped into a prompt. Cerberus computes deterministic incident facts first — groups spans by trace, flags errors and token/cost spikes, ranks worst-first — and only then hands those facts to the LLM. The model's job is to explain, not to detect. So you get answers like:

Two of the last five runs failed at the judge and agent.run steps, each flagged
error, token_spike, cost_spike (trace_id 5287c917… and 8e112c9a…). Each failed
run used 4,080 tokens and cost $0.122 — roughly 10× the 380 tokens and $0.0114
of the successful runs.
Enter fullscreen mode Exit fullscreen mode

Real trace IDs, real numbers, and it says so when the facts don't cover something.

Closing the loop

Explaining an incident is half of it. Cerberus then writes the guardrail back into SigNoz: signoz_create_alert creates a TRACES_BASED threshold rule over max(gen_ai.usage.input_tokens) — the runaway prompt that drives the cost spike — wired to a notification channel. One click. Here it is live in SigNoz's Alerts UI:

[screenshot: the armed alert in SigNoz]

That's the whole point: Cerberus never leaves SigNoz. It reads through the MCP server and writes through the same MCP server. Observe → Explain → Prevent, inside one platform.

What I learned

  • The MCP server is the best part of SigNoz for building agents on — but treat its payloads as things to discover, not things to trust from docs. Capture real responses and pin them in tests.
  • Query Builder v5 is the escape hatch whenever a convenience tool doesn't return the field you need.
  • Determinism before the LLM is what makes the answers trustworthy — the model explains facts it can't fabricate.

Cerberus is MIT-licensed and reproducible with one Foundry command. Built with AI coding assistance (Claude), as the hackathon allows.

Live: https://cerberus-sre.vercel.app
Code: https://github.com/CodeMuscle/cerberus

Top comments (0)