AI SRE Agents on SigNoz MCP: Root-Cause Analysis for $0.0013

Harjapan Singh — Sat, 25 Jul 2026 21:25:59 +0000

🏆 Built for the WeMakeDevs × Agents of SigNoz Hackathon 2026

· GitHub: github.com/HARJAPAN2005/MIB-Men-in-Backend
· Demo: https://youtu.be/sMqGD6IibTI
· Hackathon: wemakedevs.org/hackathons/signoz

How we built a reactive + proactive AI SRE platform on top of SigNoz MCP — and what we learned when 38% of our own top metrics were costing money for nothing.

⏱️ 12 min read · 🔧 Python · FastAPI · SigNoz MCP · OpenTelemetry · Foundry

Project at a Glance

We built MIB (Men in Backend) — two autonomous AI SRE agents running on top of the SigNoz MCP Server.

	Agent J	Agent K
Role	Incident responder	Observability auditor
Trigger	Alert webhook	Scheduled / on-demand
Output	Slack root-cause card	Slack precinct report + Approve button
Cost	$0.0013 per investigation	—
Key finding	Full RCA in 3.7 seconds	38% of top metrics never read by anyone

Both agents instrument themselves with OpenTelemetry GenAI semantic convention spans, emitting into the same SigNoz — so SigNoz ends up watching the agents that watch SigNoz.

The Problem with "Chat With Your Traces"
The Idea: An Autonomous Observability Workforce
Architecture
Building It: The Parts That Mattered
- Agent J's Playbook: Why Determinism Wins
- The Double-Decode Problem
- The Host-Rewrite Problem
- Foundry Patches: Deliberate Molding
- The GenAI Semantic Convention Spans
- Agent K: The Auditor Who Never Auto-Applies
- The Predictive Engine
- The Confidence Engine
SigNoz Integration: What We Actually Used
The Mistakes We Made
Lessons Learned
The Part We Didn't Expect
Conclusion

The Problem with "Chat With Your Traces"

The most common AI observability demo goes like this: connect an LLM to your telemetry backend, ask it a question, get an answer. "What's the error rate on checkout?" "Which service is slowest?" It's impressive the first time. By the fifth time, a question forms:

Why does a tool that can see everything still need me to ask?

The deeper problem is that observability has always been reactive. Dashboards show you what already happened. Alerts fire after a threshold is crossed. PagerDuty wakes someone up who then manually traces from metric → trace → log, building an evidence chain at 2 AM that any deterministic program could assemble automatically.

A chatbot doesn't fix this. A chatbot is still reactive — it just answers questions faster. What we wanted was something different: agents that act without being asked, that audit things you forgot to look at, that observe themselves so you can measure their cost, and that flag problems before the alert fires.

That's MIB.

The Idea: An Autonomous Observability Workforce

The mental model that unlocked the design was simple: treat your observability platform as a workplace, not a database.

SigNoz holds everything — traces, logs, metrics, dashboards, alerts — and the SigNoz MCP server exposes all of it through a structured tool interface. So instead of building a chatbot that queries it, we built employees who use those tools on a fixed schedule, with defined job descriptions, human-approval gates on anything destructive, and expense reports (token cost per investigation).

Two agents:

Agent J (the rookie who gets paged): alert fires → J runs a fixed 5-step MCP playbook → posts a root-cause card to Slack with one-click links back to the exact trace.
Agent K (the veteran who inspects the precinct): scheduled audits of the observability stack itself — which metrics cost money but nobody reads, which alerts fire into a dead channel — then proposes one fix, approval-gated.

And the twist: both agents instrument themselves with OpenTelemetry GenAI semantic convention spans, emitting into the same SigNoz. You end up with SigNoz watching the agents that watch SigNoz. We called this layer the Neuralyzer.

Architecture

demo-lite app ──OTLP──► SigNoz (Foundry: docker/compose)
                            │ alert webhook             ▲
                            ▼                           │ OTel (GenAI spans + metrics)
                    Agency HQ (FastAPI :8100)           │
                    ├── Agent J (5-step MCP playbook) ──► SigNoz MCP :8000
                    ├── Agent K (audit checklist)     ──►
                    └── Mission Control++ (predictive, replay, debate, postmortem)
                            │
                            ▼
                    Slack: incident cards · precinct reports · Approve buttons

Three observability layers in the same SigNoz instance:

Layer	What it is	What SigNoz sees
1 — Victim app	demo-lite, 14 microservices	OTLP traces, logs, metrics
2 — The agents	FastAPI + manual GenAI spans	`agent_j.investigation.*` spans, cost metrics
3 — MCP server	signoz-mcp itself	`mcp.tool.calls`, `mcp.tool.call.duration.*`

Mission Control++: live agent activity, predictive incident warning, and evidence graph.

Key Components

SigNoz + Foundry — We deploy SigNoz via Foundry, SigNoz's own declarative installer. The casting.yaml file is 47 lines and contains three deliberate RFC 6902 JSON patches on the generated compose file — more on those in a moment.

SigNoz MCP — The MCP server exposes 41 tools covering every layer of SigNoz: traces, logs, metrics, dashboards, alerts, and notification channels. The agents use it as their only interface to the platform.

Agency HQ — A FastAPI application that receives alert webhooks from SigNoz's Alertmanager, dispatches investigations to Agent J as background asyncio tasks (so the webhook returns fast), and runs Agent K on demand. It also hosts the Mission Control++ web UI and the SSE event stream.

Mission Control++ — A separate module (mission_control_plus.py, ~655 lines) implementing a predictive incident engine, a replay system, a confidence engine with delta attribution, an AI debate layer, evidence graph generation, and auto-postmortem generation in Markdown and PDF — no external dependencies beyond the standard library and httpx.

Building It: The Parts That Mattered

Agent J's Playbook: Why Determinism Wins

The hardest design decision was also the first: should Agent J be a true agentic loop, choosing its own tools, or a fixed deterministic playbook?

We chose fixed. Here's the reasoning:

An agentic loop that picks tools based on intermediate results produces different investigation paths for the same incident. That's interesting academically and catastrophic in a demo. More importantly, an SRE investigating a service degradation doesn't have infinite investigative freedom — there's a canonical order of operations:

Confirm the alert fired
Find the bad traces
Drill into the worst trace
Correlate the logs
Measure the metric delta

Every good SRE does this. We encoded it.

The result is run_playbook() in agent_j.py — 150 lines, five sequential MCP calls. The LLM comes in exactly once, for synthesis only:

resp = await litellm.acompletion(
    model=AGENT_J_MODEL,
    messages=[{"role": "system", "content": SYS_PROMPT},
              {"role": "user", "content": user}],
    temperature=0, max_tokens=1600,
    response_format={"type": "json_object"},
    reasoning_effort="disable",  # gemini-2.5-flash is a thinking model; disable it so
                                 # the JSON never truncates and latency stays ~2s not ~16s
)

That reasoning_effort="disable" line came from a real bug. On the first fully-instrumented run, the LLM call took 16 seconds and returned truncated JSON. The cause: gemini-2.5-flash is a thinking model; with max_tokens=1600 and thinking enabled, it was burning most of the budget on chain-of-thought reasoning before outputting any JSON, then truncating. Setting reasoning_effort="disable" cut latency to ~2s and the parse failure rate to zero.

Every MCP call also passes a searchContext argument — this isn't decoration. The MCP server uses it to tag its own telemetry, which is how the layer-3 panel on our surveillance dashboard works:

s2 = await mcp_call(client, "signoz_search_traces",
                    {"service": svc, "error": True, "start": start_ms, "end": end_ms,
                     "limit": 25, "searchContext": ctx + " (step 2: error traces)"})

mcp.tool.calls by gen_ai.tool.name shows a per-tool breakdown, and searchContext gives each call a human-readable label in the trace.

Agent J's incident card — root cause, evidence chain, and one-click trace link, generated in ~3.7 seconds.

The Double-Decode Problem

MCP tool results are not what you'd expect. The response envelope from tools/call looks like this:

{
  "result": {
    "content": [{"type": "text", "text": "{\"data\": {...}}"}]
  }
}

The actual payload is a JSON string inside result.content[0].text — every MCP call needs double-decoding. Some tools also append a [Decisions applied] human-readable block after the JSON, which breaks a naive json.loads(). Our _loads_lenient() handles both:

def _loads_lenient(text: str):
    try:
        return json.loads(text)
    except json.JSONDecodeError:
        # extract the first balanced {...} — trims the trailing [Decisions applied] block
        start = text.find("{")
        end = text.rfind("}")
        if start != -1 and end > start:
            try:
                return json.loads(text[start:end + 1])
            except json.JSONDecodeError:
                pass
        return {"_text": text}

The Host-Rewrite Problem

Every webUrl returned by MCP tools is built from the MCP server's own SIGNOZ_URL env variable, which defaults to http://signoz-signoz-0:8080 — the internal Docker service name. Not browser-clickable. We run a deep_rewrite() pass on every MCP result before using any URL:

INTERNAL_HOST = "signoz-signoz-0:8080"
BROWSER_HOST = os.environ.get("SIGNOZ_BROWSER_HOST", "localhost:8080")

def deep_rewrite(obj):
    if isinstance(obj, dict):
        return {k: deep_rewrite(v) for k, v in obj.items()}
    if isinstance(obj, list):
        return [deep_rewrite(x) for x in obj]
    return rewrite_host(obj)

The alert webhook's externalURL field is already browser-clickable (it comes from Alertmanager, not the MCP server), so we use it as-is for the alert rule deep link in the Slack card.

Foundry Patches: Deliberate Molding

One of Foundry's strengths is the ability to apply RFC 6902 JSON patches to the generated compose file without touching it directly. We used three:

Patch 1 — Healthcheck fix

The signoz-mcp image is distroless (no shell, no wget, no curl). The default healthcheck is wget --spider /livez. This can never run — the container reported as unhealthy while actually serving traffic. We patched it out:

- op: replace
  path: /services/signoz-mcp/healthcheck
  value:
    test:
      - NONE

Patch 2 — MCP self-telemetry (the third observability layer)

The MCP server logs "OpenTelemetry export not configured" by default. Two environment variables flip it on:

- op: add
  path: /services/signoz-mcp/environment/-
  value: OTEL_EXPORTER_OTLP_ENDPOINT=http://signoz-ingester-1:4317
- op: add
  path: /services/signoz-mcp/environment/-
  value: OTEL_SERVICE_NAME=signoz-mcp

After this patch, mcp.tool.calls and mcp.tool.call.duration.* metrics start appearing in SigNoz, tagged with gen_ai.tool.name. Four lines to add a complete third observability layer.

Patch 3 — Alert latency

Alertmanager's default group_wait is 30 seconds. With a 30s evaluation window on the ruler, worst-case fault-to-Slack-card is ~3.5 minutes. We patched it to 5 seconds:

- op: add
  path: /services/signoz-alertmanager/environment/-
  value: ALERTMANAGER_GROUP_WAIT=5s

This was the difference between a demo where you wait three minutes and one where the card lands before the coffee is poured.

The GenAI Semantic Convention Spans

We chose manual instrumentation over OpenLLMetry's auto-instrument for one reason: determinism. With auto-instrument, the exact attributes depend on the library version. With manual spans, we know exactly what lands in SigNoz:

with tracer.start_as_current_span("agent_j.synthesize") as span:
    span.set_attribute("gen_ai.system", get_gen_ai_system_provider(AGENT_J_MODEL))
    span.set_attribute("gen_ai.operation.name", "chat")
    span.set_attribute("gen_ai.request.model", AGENT_J_MODEL)
    span.set_attribute("gen_ai.response.model", str(resp_model))
    span.set_attribute("gen_ai.usage.input_tokens", in_tok)
    span.set_attribute("gen_ai.usage.output_tokens", out_tok)
    span.set_attribute("gen_ai.usage.cost", cost)   # computed, not from the API

gen_ai.usage.cost isn't returned by the API — we compute it from a pricing table:

GENAI_PRICING = {
    "gemini/gemini-2.5-flash": (0.30, 2.50),   # USD per 1M tokens (input, output)
    "gpt-4o-mini": (0.15, 0.60),
    "anthropic/claude-haiku-4-5-20251001": (1.00, 5.00),
}

The cost panel on the Agency Surveillance dashboard is sum(gen_ai.usage.cost) GROUP BY investigation.id in SigNoz Query Builder v5.

Real run against the redis fault: 1,733 input + 305 output tokens with gemini-2.5-flash = $0.0012824

The investigation.id attribute (carried as a Python contextvar so it flows to every child span automatically) lets you group a whole investigation in one query — the root span, all five MCP call spans, and the synthesis span share the same fingerprint.

A complete Agent J investigation as a SigNoz trace. Five MCP tool call spans, one synthesis span, gen_ai.usage.cost = 0.00128.

Agent K: The Auditor Who Never Auto-Applies

Agent K runs a four-step audit checklist via MCP:

Step A — Top metrics by ingestion volume (signoz_get_top_metrics)
Finds your cost hogs.

Step B — Cardinality check (signoz_check_metric_cardinality)
On the top-cost metric, checks how many distinct label value combinations exist. High cardinality = exponential time-series = exponential cost.

Step C — Unused metrics (signoz_check_metric_usage)
Checks whether any dashboard or alert references each top-cost metric.

On our stack: 15 of the top 39 metrics were referenced by zero dashboards and zero alerts. That's ~38.5% of top-metric ingestion going nowhere.

Step D — Orphan alerts (signoz_list_alert_rules + signoz_list_notification_channels)
Finds alert rules whose notification channels don't exist.

The check_metric_usage result has a parsing gotcha: the metric→usage map is at the top level of the JSON, not under .data. We learned this from a failed read-back that confidently reported "no unused metrics":

# WRONG:  umap = usage["data"]  ← this is None for this tool
# CORRECT:
umap = usage.get("data") if isinstance(usage.get("data"), dict) else usage
umap = {k: v for k, v in (umap or {}).items()
        if isinstance(v, dict) and ("dashboards" in v or "alerts" in v)}

The single proposed fix is approval-gated. Agent K never auto-applies anything. The Slack card has an Approve button linking to GET /agent-k/approve/{fix_id}. When clicked, K calls signoz_create_dashboard (an MCP write tool), then calls signoz_list_dashboards and confirms the title appears. If it does, the fix is confirmed.

One detail worth sharing: the pending fix is persisted to disk (/tmp/pending_fixes.json) so that if Agency HQ restarts between audit and approval, the button still works. Without this, every receiver restart invalidated all outstanding Approve buttons — which we discovered during a demo rehearsal.

Agent K's Precinct Report — 15 top-cost metrics with zero references. One click to fix.

The Predictive Engine

The Mission Control++ predictive engine runs in a background asyncio loop every 25 seconds. For each watched service it:

Fetches the last 12 minutes of traces and ERROR logs via MCP
Computes a snapshot: p95 latency, error rate, retry count, queue depth signal, throughput
Scores the trend:

Signal	Condition	Points
p95 latency	Increasing >20%	+24
Retry count	>30% increase or >8 retries	+18
Error rate	Rising >25% or >8%	+19
Throughput	Dropping >18%	+12
Historical similarity	Match found	+6 to +22

If total score ≥ 55 → emit prediction: confidence = min(97, max(66, score)), eta = max(3, 24 - confidence/5) minutes

When a real incident is investigated and its prediction was already open, the prediction transitions from open to realized. No ML model — just trend scoring against live telemetry, every 25 seconds.

The Confidence Engine

After every investigation, the confidence engine runs five scoring dimensions with additive deltas:

score = 22  # base
if traces >= 5:
    score += 18   # "Trace matched"
if logs >= 1:
    score += 22   # "Metric correlation"
if delta_x >= 1.4:
    score += 14   # "Retry spike"
if prediction and prediction.get("historical_similarity"):
    score += 11   # "Historical similarity"
if verdict.get("_fallback"):
    score -= 12   # "Fallback verdict penalty"

Each delta has a label that appears in the Mission Control UI and in the auto-generated postmortem. The result is a confidence score with an audit trail — you can see why it's 76%, not just that it is.

Auto-generated postmortem, produced seconds after the investigation completes. Downloadable as Markdown or PDF.

Investigation replay in Mission Control++ — each step shows the tool called, timestamp, and evidence count.

SigNoz Integration: What We Actually Used

Capability	How MIB Uses It
Traces	Error trace search; worst-trace drill-down; predictive snapshot
Logs	Service ERROR logs (root cause messages) + trace-correlated logs
Metrics	Pre/in-window call-volume delta; predictive snapshots
Dashboards (read)	Agent K read-back to confirm fix applied
Dashboards (write)	Agent K fix via `signoz_create_dashboard` — an MCP write tool
Alerts (read)	Agent K step D: list rules, check channels
Alerts (create)	`setup_alerts.sh` creates both alert rules via MCP
Notification channels	Created via MCP; Agent K checks for orphans
Alert webhooks	Alertmanager v4 payload → Agency HQ `/webhook/alert`
Deep links	`webUrl` from traces/alerts → Slack card one-click navigation
OTel traces (self)	Agency HQ auto-instrumented; manual GenAI semconv spans
OTel metrics (self)	`mib.investigations.total`, `mib.investigation.cost`, etc.
MCP self-telemetry	`mcp.tool.calls` by `gen_ai.tool.name` — third observability layer
Query Builder v5	6-panel Agency Surveillance dashboard
Foundry	Full deployment + 3 RFC 6902 patches

Agency Surveillance dashboard — cost per investigation, token usage, latency p95, and MCP server self-telemetry (the third observability layer).

One Query Builder v5 gotcha worth calling out: aggregation queries cannot orderBy the raw metric name. If you try orderBy: mcp.tool.calls on a sum(mcp.tool.calls) query, SigNoz rejects it with "invalid order by key for an aggregation query." You must orderBy the aggregation expression itself, or a groupBy key. We fixed this by ordering on gen_ai.tool.name.

The Mistakes We Made

Real projects have real failure modes. Here are the six that cost us the most time:

🔴 Mistake 1: Docker Desktop on Windows

ClickHouse Keeper segfaults (exit 139) under Docker Desktop because of its virtualization layer. The fix is native Docker Engine inside a real WSL2 distro.

Lost time: 45 minutes at the start of the hackathon.

🔴 Mistake 2: WSL Idle Teardown

After fixing Docker, the entire SigNoz stack started restarting every 2–3 minutes. Diagnosis: no wsl.exe session was held open, so WSL was idling and tearing down the distro — stopping Docker, stopping all containers.

The fix: a Windows Scheduled Task that runs at logon to keep the distro pinned:

wsl -d Ubuntu-24.04 -u root -- sleep infinity

🔴 Mistake 3: Assuming MCP Uses SSE

The SigNoz MCP server's HTTP transport returns plain JSON, not Server-Sent Events. Our first call to tools/list returned empty because we were grepping for ^data: SSE lines in the response. The server is also stateless — no session ID needed.

🔴 Mistake 4: Wrong Log Query for Root Cause

The first attempt at Step 4 correlated logs by trace_id. This surfaced a success log — "Order placed successfully" — as the root cause. Switching to service-scoped severity=ERROR logs captured dial tcp ...:6379: i/o timeout immediately.

Rule: Lead with ERROR-severity logs. Use trace correlation for context, not primary evidence.

🔴 Mistake 5: Thinking Model Truncation

gemini-2.5-flash with thinking enabled burned most of max_tokens=1600 on reasoning before writing the JSON. The JSON truncated. response_format=json_object alone wasn't enough.

Adding reasoning_effort="disable" fixed both truncation and latency: 16s → 2s.

🔴 Mistake 6: In-Memory Approval State

Agent K's pending fixes were stored only in memory. Every Agency HQ restart wiped them, making all Approve buttons return 404. We added disk persistence and idempotent read-back to /tmp/pending_fixes.json.

Lessons Learned

Deterministic playbooks beat agentic loops for reliability.
In a production SRE context, you want reproducible investigations. A fixed playbook means the same alert always runs the same tools in the same order. You can test it, measure its latency, and know exactly what it costs.

The third observability layer is free if you think to enable it.
Getting the MCP server to emit its own traces and metrics required four lines in casting.yaml. The result is a dashboard panel showing MCP tool call latency broken down by tool — signoz_get_trace_details averages 244ms, signoz_query_metrics averages 328ms.

Self-observability is a competitive moat.
If your AI system can tell you how much it costs, how long it takes, and which operations are slowest, it becomes auditable in a way that black-box systems cannot. The $0.0013 headline only exists because we emit gen_ai.usage.cost on every synthesis span.

"Human-gated" is not a weakness.
Agent K's approval requirement is often the most impressive part of the demo. It shows the design is aware of its own failure modes.

What I'd redesign: The predictive engine scores trends linearly and doesn't account for time-of-day seasonality. The evidence graph is hardcoded to the redis→cart→checkout→payment chain; a production version would derive topology from actual trace parent-child spans.

What's next: True agentic investigation (J decides mid-playbook to dig deeper on a specific span), multi-service correlation across a dependency chain, and Slack interactive messages for direct in-chat approval.

The Part We Didn't Expect

The most surprising result wasn't the AI verdict quality or the MCP integration. It was what happened when Agent K looked at unused metrics in a real system.

15 of our top 39 metrics were being ingested and never read.

Not because the system was broken — because the dashboards and alerts had evolved without anyone going back to prune the metrics. The data was being stored, compressed, and replicated in ClickHouse on every scrape, and nobody was looking at any of it.

That's not an observability problem. That's an organizational problem that observability makes visible.

Agent K's job isn't just to find the issue — it's to surface the cost in a language everyone understands: "~38.5% of your top-metric ingestion is unread." And then propose a fix that a human approves. That's the loop that makes AI agents useful in practice: not autonomous action, but autonomous discovery with human-controlled execution.

Conclusion

We set out to build two AI SREs and ended up with something more interesting: a system where the tool your agents use to observe your stack also observes them back.

Three layers of observability. One SigNoz. $0.0013 per investigation.

The MCP protocol made this possible in a way that wasn't true a year ago. When the interface to your observability platform is a clean set of tool definitions — reads and writes, traces and dashboards and alerts — the agents can do SRE work, not just SRE queries. That's a different category of useful.

→ Code: github.com/HARJAPAN2005/MIB-Men-in-Backend — three commands to deploy.

Built with heavy use of AI assistants (Claude, Gemini) — declared per hackathon rules. See README §AI Usage Declaration.

MIB — Men in Backend · WeMakeDevs × Agents of SigNoz Hackathon · Jul 20–26, 2026

DEV Community: Harjapan Singh