Vickyavh7

Posted on Jul 26

Incident Sentinel: An SRE Copilot That Investigates SigNoz Alerts — and Observes Itself

#signoz #ai #opentelemetry #observability

When an alert fires at 2am, the expensive part isn’t the paging. It’s the scavenger hunt.

Which service broke?

Which trace proves it?

Which log line explains it?

What changed now?

Incident Sentinel is an SRE incident copilot that investigates SigNoz alerts through the SigNoz MCP server, posts an evidence-backed report (and postmortem) to Slack, and — the part that matters for real ops — observes itself in the same SigNoz instance.

The agent that debugs your system should not be a black box.

Repo: github.com/Vickyavh7/signoz-sentinel

The observability gap (one layer up)

Modern on-call is no longer only “watch the checkout API.” AI agents are entering the loop: they call tools, reason over telemetry, and write summaries. Observability has to cover both layers:

Layer	What you need to see
The outage	Errors, latency, traces, logs across services
The investigator	Tool calls, LLM latency, tokens, USD cost, accuracy

Without the second layer, teams often have:

no investigation span
no token count
no cost signal
no way to say “how often is this right?”

That’s the same “flying blind” story we tell about LLM apps — applied to the incident agent itself.

What we built

checkout → payment → inventory     (demo apps + fault injection)
        │ OTLP
        ▼
   SigNoz + MCP
        │ alert webhook
        ▼
   Incident Sentinel
        ├── MCP tools (traces, logs, metrics, alerts)
        ├── LLM investigation loop
        ├── Slack report + markdown postmortem
        └── OTLP → same SigNoz (tokens, cost, agent quality)

Concrete pieces:

SigNoz as the single pane — traces, metrics, logs, alerts, dashboards.
Fault-injectable demo chain on Kubernetes: checkout-api → payment-svc → inventory-svc.
Incident Sentinel — FastAPI webhook + tool-calling agent over SigNoz MCP.
Slack reports with evidence links back into SigNoz.
Auto-postmortems — timeline, impact, evidence, cost; also served at GET /postmortems/{ruleId}.
Self-telemetry — investigation spans, GenAI chat spans, tool spans, sentinel_tokens_total, sentinel_cost_usd_total.
Measured accuracy — real fault evals in evals/, not vibes.

We kept the core agent custom so we own GenAI attributes and cost metrics end-to-end.

Here’s the live service map during a fault window — demo apps plus the copilot as a first-class service:

Figure 1. SigNoz Services — the outage services and incident-sentinel in one pane.

How an investigation works

A SigNoz alert fires (errors, latency, or our cost meta-alert).
The webhook hits Incident Sentinel (acks fast; work continues in the background).
The LLM chooses MCP tools — search traces, pull trace details, search logs, query metrics.
It correlates across signals and follows the dependency chain.
It concludes with summary, root cause, impact, timeline, and suggested follow-ups.
Slack gets the report; a markdown postmortem is generated.
The whole run is traced back into SigNoz as service incident-sentinel.

We intentionally do not auto-remediate. The copilot diagnoses; humans keep the execute button. That boundary is a feature for trust, not a missing checkbox.

Alert rules are the entry point — including a meta-alert on the copilot’s own spend:

Figure 2. Alert Rules — service outages plus sentinel-cost-budget (kind: meta-alert).

And every investigation leaves a real OpenTelemetry trace the on-call can open:

Figure 3. A sentinel.investigate trace (~50s) — LLM steps and MCP tool calls, not a black box.

A real postmortem excerpt (auto-generated during a checkout error spike):

# Postmortem: checkout-error-spike
- Primary service: checkout-api · Confidence: 0.8
- Investigation duration: 192.3s · Cost: $0.0112

## Summary
checkout-api error spike due to downstream service failure

## Root cause
Origin appears to be checkout-api (cascade-refined).
inventory-svc failure, possibly due to a recent deployment…

## Evidence
- checkout-api trace with error — GET /checkout → 502

Design decision: the watcher is watched

We put a cost-budget meta-alert on sentinel_cost_usd_total.

If the copilot spends too much investigating, it pages you.

This isn’t a slideware idea. During testing, a few investigations with a larger hosted model blew through a tiny demo budget. SigNoz fired sentinel-cost-budget, routed it to the copilot’s own webhook, and the copilot investigated its own cost overrun and posted that report to Slack.

Full circle: the incident agent became the incident.

Operations telemetry for the copilot itself:

Figure 4. Copilot Operations — investigation volume, GenAI calls, and MCP tool usage.

That is the observability use case in one sentence: instrument the outage and the investigation on the same OpenTelemetry-native backend.

Measuring the agent (instead of trusting it)

The common gap in AI-for-SRE systems: nobody asks how often the agent is right.

Our eval harness (evals/run_evals.py) injects real faults:

Scenario	Fault	What “pass” means
checkout-errors	error spike on checkout/payment	names the right service + failure mode
payment-latency	latency-only on payment	names payment + latency language
inventory-errors	inventory failures (cascade)	names inventory, not a mid-chain symptom

Scoring is strict: echoing the alert name in prose doesn’t count. The verdict’s service field must be correct.

Latest run: 3/3 (see evals/RESULTS.md).

The first honest failures taught us more than the passes:

The agent once blamed incident-sentinel itself for a latency incident — because its own spans live in the same SigNoz it queries. Self-observability can pollute self-investigation unless you forbid it.
Cascade faults made it blame mid-chain services (e.g. payment) when the origin was inventory. We added cascade refinement: never blame upstream of the alerted service; prefer the deepest erroring hop in checkout → payment → inventory.

Eval metrics are also exported to SigNoz (sentinel_eval_*) and shown on an Agent Quality dashboard — accuracy as a first-class observability signal:

Figure 5. Agent Quality — cost, investigation volume, and eval pass ratio in SigNoz.

What broke while building

Infrastructure and auth are part of any real observability setup:

Older Linux hosts and tooling glibc mismatches — we ran Foundry tooling in a Debian container with the host Docker socket.
ClickHouse Keeper instability on one image pin; pinning an older keeper image stabilized the stack.
Auth footguns: Service Account API keys vs user JWTs are not interchangeable with MCP. Wrong header → 401/403 that looks like “MCP is broken.”
Big models are slow: webhook delivery must ack immediately; investigations run in the background with timeouts and retries.
Graceful degradation: if the LLM never returns a valid conclusion, the report is built from MCP evidence only and marked degraded — no canned fake root cause.

Guardrails that make this ops-shaped

Dedupe window per alert / rule
Concurrency cap and bounded investigation steps
Scoped SigNoz Service Account key in a Kubernetes Secret
Token + USD cost + duration on every run
Evidence-only fallback when the model fails

Enough to be credible in a lab. Not a claim that this is production SaaS tomorrow.

Try the flow

./demo/break.sh errors
Alert (or POST /investigate) → Slack report + postmortem
SigNoz → service incident-sentinel → investigation trace
Dashboards: Copilot Operations + Agent Quality
Check evals/RESULTS.md
Point at sentinel-cost-budget — the watcher is watched

Takeaways

Observability for AI agents isn’t only “trace the LLM app.” Sometimes the agent is the SRE workflow.
Instrument both sides — the outage and the investigation — on one OpenTelemetry-native backend.
Measure accuracy with real faults, or you’re shipping confidence, not competence.
Meta-alerts on agent cost turn self-telemetry into something operators can actually page on.

DEV Community