I gave an AI SRE copilot write access to SigNoz — here's what the MCP server actually returns

Ganesh J — Sun, 26 Jul 2026 14:20:58 +0000

AI agents fail in ways classic monitoring misses. A tool call silently retries. A prompt balloons your token bill 4×. A judge step times out. An agent quietly loops. Your observability platform has all of it — but staring at a trace waterfall at 2am isn't the same as knowing what broke.

So for WeMakeDevs' Agents of SigNoz, I built Cerberus: an AI SRE copilot that reads your agent's telemetry back out of SigNoz, explains the incident in plain English with trace citations, and then writes the alert rule that would have caught it — all through SigNoz's own MCP server.

Live demo: https://cerberus-sre.vercel.app
Code: https://github.com/CodeMuscle/cerberus

The idea: emit and consume

Most observability projects only emit telemetry — they push spans and stop. Cerberus does the agent-native thing: it emits, then consumes its own agent's traces back and acts on them. Three heads, three jobs:

Observe — every agent run lands in SigNoz as OpenTelemetry traces, with token usage, cost, latency and errors per step.
Explain — a copilot reads the ranked incidents and answers "what just went wrong?" grounded only in the facts, citing the exact trace_id.
Prevent — one click writes the SigNoz alert rule for the incident's root cause.

Why MCP, not the REST API

The interesting decision was the read path. SigNoz ships an MCP server — the same tool surface an AI client gets: signoz_execute_builder_query, signoz_search_traces, signoz_create_alert, and ~40 more. Instead of scraping a REST endpoint, Cerberus speaks MCP. That means the exact interface an LLM would use is the interface the product is built on.

Deploying it is one command, because SigNoz's Foundry installer brings up the whole stack plus the MCP server from a single casting file:

curl -fsSL https://signoz.io/foundry.sh | bash
foundryctl cast -f casting.yaml     # SigNoz on :8080, MCP on :8000

casting.yaml and its lockfile are committed, so the deployment is reproducible — anyone can re-run it.

The part nobody tells you: the payload is not the docs

Here's what actually cost me a day, and what I'd want another builder to know.

1. signoz_search_traces returns a fixed column projection — with no gen_ai.* attributes. The obvious tool for "give me the spans" hands back a fixed set of columns. My token and cost data — gen_ai.usage.input_tokens, gen_ai.usage.cost_usd — came back empty every time. The whole product is about token and cost spikes, and they were zero.

The fix was to drop to the Query Builder v5 tool and select those columns explicitly:

{"name": "gen_ai.usage.input_tokens", "fieldDataType": "number",
 "signal": "traces", "fieldContext": "tag"}

signoz_execute_builder_query lets you name exactly the attributes you want. Once I selected the gen_ai.* tags by hand, the real numbers showed up.

2. The response carries a human-readable note after the JSON. The payload looks like this:

{ ...valid JSON... }
note: returned 3 rows (limit 3) — more results likely exist (hasMore=true).

A naive json.loads() throws on the trailing text — and if you catch that broadly, you silently get zero spans and no error. The fix is to decode a prefix (JSONDecoder().raw_decode) instead of parsing the whole string.

3. Rows nest one level down, and an empty window returns null. Each row wraps its fields under a data sub-object alongside a timestamp, and an empty time window comes back as "rows": null rather than [] — which will happily crash a flat-map if you assume a list. One or [] guard fixes it.

None of these are in the docs. All three are captured as tests against real captured payloads, so they don't regress.

Grounded, not hallucinated

The copilot doesn't get raw traces dumped into a prompt. Cerberus computes deterministic incident facts first — groups spans by trace, flags errors and token/cost spikes, ranks worst-first — and only then hands those facts to the LLM. The model's job is to explain, not to detect. So you get answers like:

Two of the last five runs failed at the judge and agent.run steps, each flagged
error, token_spike, cost_spike (trace_id 5287c917… and 8e112c9a…). Each failed
run used 4,080 tokens and cost $0.122 — roughly 10× the 380 tokens and $0.0114
of the successful runs.

Real trace IDs, real numbers, and it says so when the facts don't cover something.

Closing the loop

Explaining an incident is half of it. Cerberus then writes the guardrail back into SigNoz: signoz_create_alert creates a TRACES_BASED threshold rule over max(gen_ai.usage.input_tokens) — the runaway prompt that drives the cost spike — wired to a notification channel. One click. Here it is live in SigNoz's Alerts UI:

[screenshot: the armed alert in SigNoz]

That's the whole point: Cerberus never leaves SigNoz. It reads through the MCP server and writes through the same MCP server. Observe → Explain → Prevent, inside one platform.

What I learned

The MCP server is the best part of SigNoz for building agents on — but treat its payloads as things to discover, not things to trust from docs. Capture real responses and pin them in tests.
Query Builder v5 is the escape hatch whenever a convenience tool doesn't return the field you need.
Determinism before the LLM is what makes the answers trustworthy — the model explains facts it can't fabricate.

Cerberus is MIT-licensed and reproducible with one Foundry command. Built with AI coding assistance (Claude), as the hackathon allows.

Live: https://cerberus-sre.vercel.app
Code: https://github.com/CodeMuscle/cerberus

Building an AI research copilot that catches its sources lying

Ganesh J — Mon, 06 Jul 2026 07:00:24 +0000

Research tools forget across sessions, and they never notice when two sources disagree. Crosscheck is a small copilot on top of cogneethat does both: persistent memory of everything you feed it, and a hero feature that flags when sources contradict each other — e.g. "FooDB sustained 50,000 req/s" (2021) vs "only 10,000 req/s" (2024).

The obvious design — and why it wasn't enough

The first instinct is to make contradiction detection a pure graph query: build cognee's knowledge graph, then look for the same entity+attribute with different values across sources. Elegant, but it breaks on a real local stack (llama3.1:8b via Ollama). Two failures show up:

The knowledge graph flattens quantities. The extractor turns "50,000 requests per second" into a generic requests per second node and drops the number. The conflicting values never make it into the graph.
Entities get merged across sources. Every mention of FooDB collapses to one node, so the two throughput claims dedup into a single edge — nothing left to compare.

So the graph is great for storage, visualization, and cited retrieval, but it can't be the source of truth for a quantitative contradiction.

The fix: extract faithful claims, judge them structurally

Crosscheck reads claims straight from each source's raw text — a thin, flat (subject, predicate, object) extraction that keeps the number verbatim and tags it with the source id and timestamp. Extracting one value is easy even for a small local model, unlike a full graph schema. Then a two-stage engine:

Structural pre-filter: group claims by normalized (subject, predicate); flag pairs with the same key, different value, different source.
LLM judge: confirm each candidate actually contradicts ("cannot both be true"), with the reason.

On the FooDB pack this fires exactly once: 50k (2021) vs 10k (2024), confirmed.

Making cognee survive a weak local model

Getting the graph to build at all on llama3.1:8b took three settings, all in .env.example: switch cognee's structured-output framework to BAML (its schema-aligned parsing tolerates loose JSON), neutralize the fragile chunk summarization task (which small models can't satisfy and which Crosscheck doesn't use), and turn off multi-user access control so a direct graph read sees the whole store.

Self-improving

A thin gap finder ranks sparsely-connected nodes and asks the LLM for the next research question — so the copilot tells you what it's missing. Everything persists in cognee's stores, so a fresh process re-answers without re-ingesting.

The same engine, a second product: Argus

Once the contradiction engine existed, it turned out to be domain-agnostic — "the same fact, two sources, two different values" is a shape that shows up far beyond research. So we pointed it at money. Argus is a spend & contract leakage*auditor: a contract line ("early-pay credit due: $2,400") and an invoice line ("credit applied: $0") are just a contradiction with a dollar gap. Argus reuses Crosscheck's claims + contradictions code **unchanged* and adds one thing — a deterministic dollar-impact number on each finding. On a small demo pack it surfaces $5,300 in leakage across three issues, each with the exact documents and dates a finance team could act on. Same engine, a completely different problem — which is the real point: catching sources that disagree is a primitive, not a feature.

Runs fully offline on Ollama; an OpenAI or Gemini key is a drop-in alternative.

Repo: https://github.com/CodeMuscle/crosscheck

DEV Community: Ganesh J