DEV Community: Vickyavh7

Incident Sentinel: An SRE Copilot That Investigates SigNoz Alerts — and Observes Itself

Vickyavh7 — Sun, 26 Jul 2026 03:18:13 +0000

When an alert fires at 2am, the expensive part isn’t the paging. It’s the scavenger hunt.

Which service broke?

Which trace proves it?

Which log line explains it?

What changed now?

Incident Sentinel is an SRE incident copilot that investigates SigNoz alerts through the SigNoz MCP server, posts an evidence-backed report (and postmortem) to Slack, and — the part that matters for real ops — observes itself in the same SigNoz instance.

The agent that debugs your system should not be a black box.

Repo: github.com/Vickyavh7/signoz-sentinel

The observability gap (one layer up)

Modern on-call is no longer only “watch the checkout API.” AI agents are entering the loop: they call tools, reason over telemetry, and write summaries. Observability has to cover both layers:

Layer	What you need to see
The outage	Errors, latency, traces, logs across services
The investigator	Tool calls, LLM latency, tokens, USD cost, accuracy

Without the second layer, teams often have:

no investigation span
no token count
no cost signal
no way to say “how often is this right?”

That’s the same “flying blind” story we tell about LLM apps — applied to the incident agent itself.

What we built

checkout → payment → inventory     (demo apps + fault injection)
        │ OTLP
        ▼
   SigNoz + MCP
        │ alert webhook
        ▼
   Incident Sentinel
        ├── MCP tools (traces, logs, metrics, alerts)
        ├── LLM investigation loop
        ├── Slack report + markdown postmortem
        └── OTLP → same SigNoz (tokens, cost, agent quality)

Concrete pieces:

SigNoz as the single pane — traces, metrics, logs, alerts, dashboards.
Fault-injectable demo chain on Kubernetes: checkout-api → payment-svc → inventory-svc.
Incident Sentinel — FastAPI webhook + tool-calling agent over SigNoz MCP.
Slack reports with evidence links back into SigNoz.
Auto-postmortems — timeline, impact, evidence, cost; also served at GET /postmortems/{ruleId}.
Self-telemetry — investigation spans, GenAI chat spans, tool spans, sentinel_tokens_total, sentinel_cost_usd_total.
Measured accuracy — real fault evals in evals/, not vibes.

We kept the core agent custom so we own GenAI attributes and cost metrics end-to-end.

Here’s the live service map during a fault window — demo apps plus the copilot as a first-class service:

Figure 1. SigNoz Services — the outage services and incident-sentinel in one pane.

How an investigation works

A SigNoz alert fires (errors, latency, or our cost meta-alert).
The webhook hits Incident Sentinel (acks fast; work continues in the background).
The LLM chooses MCP tools — search traces, pull trace details, search logs, query metrics.
It correlates across signals and follows the dependency chain.
It concludes with summary, root cause, impact, timeline, and suggested follow-ups.
Slack gets the report; a markdown postmortem is generated.
The whole run is traced back into SigNoz as service incident-sentinel.

We intentionally do not auto-remediate. The copilot diagnoses; humans keep the execute button. That boundary is a feature for trust, not a missing checkbox.

Alert rules are the entry point — including a meta-alert on the copilot’s own spend:

Figure 2. Alert Rules — service outages plus sentinel-cost-budget (kind: meta-alert).

And every investigation leaves a real OpenTelemetry trace the on-call can open:

Figure 3. A sentinel.investigate trace (~50s) — LLM steps and MCP tool calls, not a black box.

A real postmortem excerpt (auto-generated during a checkout error spike):

# Postmortem: checkout-error-spike
- Primary service: checkout-api · Confidence: 0.8
- Investigation duration: 192.3s · Cost: $0.0112

## Summary
checkout-api error spike due to downstream service failure

## Root cause
Origin appears to be checkout-api (cascade-refined).
inventory-svc failure, possibly due to a recent deployment…

## Evidence
- checkout-api trace with error — GET /checkout → 502

Design decision: the watcher is watched

We put a cost-budget meta-alert on sentinel_cost_usd_total.

If the copilot spends too much investigating, it pages you.

This isn’t a slideware idea. During testing, a few investigations with a larger hosted model blew through a tiny demo budget. SigNoz fired sentinel-cost-budget, routed it to the copilot’s own webhook, and the copilot investigated its own cost overrun and posted that report to Slack.

Full circle: the incident agent became the incident.

Operations telemetry for the copilot itself:

Figure 4. Copilot Operations — investigation volume, GenAI calls, and MCP tool usage.

That is the observability use case in one sentence: instrument the outage and the investigation on the same OpenTelemetry-native backend.

Measuring the agent (instead of trusting it)

The common gap in AI-for-SRE systems: nobody asks how often the agent is right.

Our eval harness (evals/run_evals.py) injects real faults:

Scenario	Fault	What “pass” means
checkout-errors	error spike on checkout/payment	names the right service + failure mode
payment-latency	latency-only on payment	names payment + latency language
inventory-errors	inventory failures (cascade)	names inventory, not a mid-chain symptom

Scoring is strict: echoing the alert name in prose doesn’t count. The verdict’s service field must be correct.

Latest run: 3/3 (see evals/RESULTS.md).

The first honest failures taught us more than the passes:

The agent once blamed incident-sentinel itself for a latency incident — because its own spans live in the same SigNoz it queries. Self-observability can pollute self-investigation unless you forbid it.
Cascade faults made it blame mid-chain services (e.g. payment) when the origin was inventory. We added cascade refinement: never blame upstream of the alerted service; prefer the deepest erroring hop in checkout → payment → inventory.

Eval metrics are also exported to SigNoz (sentinel_eval_*) and shown on an Agent Quality dashboard — accuracy as a first-class observability signal:

Figure 5. Agent Quality — cost, investigation volume, and eval pass ratio in SigNoz.

What broke while building

Infrastructure and auth are part of any real observability setup:

Older Linux hosts and tooling glibc mismatches — we ran Foundry tooling in a Debian container with the host Docker socket.
ClickHouse Keeper instability on one image pin; pinning an older keeper image stabilized the stack.
Auth footguns: Service Account API keys vs user JWTs are not interchangeable with MCP. Wrong header → 401/403 that looks like “MCP is broken.”
Big models are slow: webhook delivery must ack immediately; investigations run in the background with timeouts and retries.
Graceful degradation: if the LLM never returns a valid conclusion, the report is built from MCP evidence only and marked degraded — no canned fake root cause.

Guardrails that make this ops-shaped

Dedupe window per alert / rule
Concurrency cap and bounded investigation steps
Scoped SigNoz Service Account key in a Kubernetes Secret
Token + USD cost + duration on every run
Evidence-only fallback when the model fails

Enough to be credible in a lab. Not a claim that this is production SaaS tomorrow.

Try the flow

./demo/break.sh errors
Alert (or POST /investigate) → Slack report + postmortem
SigNoz → service incident-sentinel → investigation trace
Dashboards: Copilot Operations + Agent Quality
Check evals/RESULTS.md
Point at sentinel-cost-budget — the watcher is watched

Takeaways

Observability for AI agents isn’t only “trace the LLM app.” Sometimes the agent is the SRE workflow.
Instrument both sides — the outage and the investigation — on one OpenTelemetry-native backend.
Measure accuracy with real faults, or you’re shipping confidence, not competence.
Meta-alerts on agent cost turn self-telemetry into something operators can actually page on.

I Already Had Grafana + Loki + Prometheus. Here's What Clicked When I Self-Hosted SigNoz Anyway.

Vickyavh7 — Tue, 14 Jul 2026 09:12:22 +0000

Three Tabs Was My Debugging Workflow. Then One Checkout Failed.

Grafana. Loki. Prometheus.

That stack already ran on my Kubernetes cluster. It was fine for dashboards and alerts.

It was not fine for the question I actually ask at 2am:

Why did this checkout fail — and what log line explains the failing span?

For the SigNoz Early Win, I self-hosted SigNoz on the same cluster, left Grafana/Loki/Prometheus alone, and pointed real OpenTelemetry traffic at SigNoz.

I expected a fourth tab.

I got a shorter path from error span → matching log.

Before / after (same broken checkout)

Step	My old stack	SigNoz
1	Find `payment failed` in Loki	Open Traces, filter `GET /checkout` + `has_error`
2	Guess service + time window	Open the waterfall — `payment-gateway` owns the duration
3	Jump to Prometheus for error rate	Read the log that already carries the same `trace_id`
4	Hope a trace backend exists	Done

That four-to-three cut is the whole post. Everything else is how I got there.

Context: Same Cluster, Existing Stack Left Alone

Cluster: Kubernetes v1.30.5, Helm v3.16, SigNoz Helm chart signoz/signoz 0.132.2.

Other workloads already lived there (pipelines, notebooks, a catalog service), plus Grafana + Loki + Prometheus.

No cloud load balancer — UIs are exposed the same way Grafana is: NodePort. SigNoz UI: 32010.

Install: Helm + NodePort (reproducible)

helm repo add signoz https://charts.signoz.io
helm repo update
kubectl create namespace signoz

helm upgrade --install signoz signoz/signoz \
  -n signoz \
  -f helm/signoz-values.yaml \
  --timeout 25m \
  --wait

helm upgrade --install signoz-k8s-infra signoz/k8s-infra \
  -n signoz \
  -f helm/k8s-infra-values.yaml \
  --wait

kubectl apply -f k8s/demo-app.yaml

Or: ./scripts/deploy.sh

Values that mattered for me:

# helm/signoz-values.yaml (excerpt)
global:
  cloud: other
  clusterName: my-cluster

clickhouse:
  persistence:
    enabled: true
    size: 20Gi

signoz:
  service:
    type: NodePort
    port: 8080
    nodePort: 32010

About five minutes later ClickHouse, ZooKeeper, signoz-0, and the OTel Collector were Running.

Grafana, Loki, and Prometheus stayed untouched.

Coexistence was the point — not a migration fantasy.

Sending Real Data: Break Checkout on Purpose

Demo FastAPI service:

/inventory, /checkout, metrics probe
~25% of checkouts fail with a nested payment-gateway span
exports traces and logs over OTLP
CronJob keeps traffic fresh

Gotcha #1: OTLP/gRPC (4317) hit transient UNAVAILABLE right after UI signup. OTLP/HTTP on 4318 worked immediately.

OTEL_EXPORTER_OTLP_ENDPOINT=http://signoz-otel-collector.signoz.svc.cluster.local:4318
# posts to .../v1/traces and .../v1/logs

Then k8s-infra DaemonSets shipped cluster metrics into the same backend.

Same-afternoon evidence

Signal	Count
Spans (`signoz-demo-api`)	3.6k+
Error spans	270+
Logs	900+
Metric series	7k+

Failing checkout shape:

GET /checkout           hasError=true   ~1203 ms
  checkout              hasError=true
    payment-gateway     hasError=true   ~1201 ms

Matching log (same trace_id):

payment failed after 1200ms

What I Touched (Early Win Checklist)

1. Traces — error list for `signoz-demo-api`

2. Trace detail — the join starts here

Waterfall for the failing GET /checkout. payment-gateway is the hop that eats the duration and carries the error.

This is the money screenshot: open this span tree, keep the trace_id, then look at logs — you should land on the payment failure without rebuilding the query by hand.

3. Logs — same failure, same `trace_id`

4–5. Metrics + dashboards

Cluster series from k8s-infra plus spanmetrics-derived latency for the demo service on a small RED-style board. Useful for “is the cluster OK?” next to “is this request OK?”

6. Alerts — `checkout-error-spike`

Notification channel first (the API refuses rules without one). Rule fired almost immediately — expected for a service that fails on purpose.

Favorite Feature: The Join I Stopped Building by Hand

Grafana + Loki + Prometheus can correlate traces and logs.

In practice my muscle memory was:

Spot a failure in logs
Guess service + window
Open another tool for traces (if one exists)
Lose the thread

In SigNoz:

Traces → GET /checkout + has_error
Waterfall → payment-gateway
Log with the same trace_id → payment failed after 1200ms

OpenTelemetry does the correlation when both signals share context. SigNoz’s UI assumes you want that join and makes the click path the default — not a custom derived field I have to remember under pressure.

One sentence for the judges:

SigNoz turned “why did this checkout die?” into a click, not a scavenger hunt across tools I already keep running.

What Broke (Tell Past-Me)

Finish SigNoz UI signup before trusting OTLP
Prefer OTLP/HTTP for a first demo
Services list can look empty while query results already have data
Alert rules need a notification channel first
Don’t rip out Grafana/Loki/Prom to “prove” SigNoz — coexistence is the better story

What I’d Do Next

Push this repo public and pin the chart version in CI
Add a dashboard screenshot + saved query for the checkout error rate
Wire a real Slack/webhook channel (not just “it fires in the UI”)
Try the same join on a multi-service path (not only the demo API)
Only then talk HA ClickHouse / auth — after the debugging loop is sticky

Thanks for Reading

If you’re doing the Early Win: install SigNoz next to whatever you already run, break one request on purpose, and write about the click that saved you a tab.

Demystifying AWS IAM: Best Practices for User Group Management and Role Creation

Vickyavh7 — Sun, 21 Apr 2024 13:45:33 +0000

Introduction: In this blog, we will explore the best practices for managing user groups, creating roles, and configuring STS policies in AWS IAM to ensure secure access to resources.

User Group Management: Create user groups to streamline permissions management and reduce the risk of errors. Follow the principle of least privilege and grant only necessary permissions to each group.

Role Creation: Define roles with specific permissions for entities like IAM users or AWS services. Adhere to the principle of least privilege, regularly review and update roles to align with security policies.

STS Policies: Define permissions for roles using STS policies to control actions and limit permissions scope. Review and test policies to ensure they align with access control rules.

Conclusion: Implementing effective user group management, role creation, and STS policy configuration enhances security, access control, and compliance in AWS IAM. Following best practices ensures a robust IAM strategy in the cloud environment.

DEV Community: Vickyavh7

Incident Sentinel: An SRE Copilot That Investigates SigNoz Alerts — and Observes Itself

The observability gap (one layer up)

What we built

How an investigation works

Design decision: the watcher is watched

Measuring the agent (instead of trusting it)

What broke while building

Guardrails that make this ops-shaped

Try the flow

Takeaways

Links

I Already Had Grafana + Loki + Prometheus. Here's What Clicked When I Self-Hosted SigNoz Anyway.

Three Tabs Was My Debugging Workflow. Then One Checkout Failed.

Before / after (same broken checkout)

Context: Same Cluster, Existing Stack Left Alone

Install: Helm + NodePort (reproducible)

Sending Real Data: Break Checkout on Purpose

Same-afternoon evidence

What I Touched (Early Win Checklist)

1. Traces — error list for `signoz-demo-api`

2. Trace detail — the join starts here

3. Logs — same failure, same `trace_id`

4–5. Metrics + dashboards

6. Alerts — `checkout-error-spike`

Favorite Feature: The Join I Stopped Building by Hand

What Broke (Tell Past-Me)

What I’d Do Next

Thanks for Reading

Links

Demystifying AWS IAM: Best Practices for User Group Management and Role Creation

DEV Community: Vickyavh7

Incident Sentinel: An SRE Copilot That Investigates SigNoz Alerts — and Observes Itself

The observability gap (one layer up)

What we built

How an investigation works

Design decision: the watcher is watched

Measuring the agent (instead of trusting it)

What broke while building

Guardrails that make this ops-shaped

Try the flow

Takeaways

Links

I Already Had Grafana + Loki + Prometheus. Here's What Clicked When I Self-Hosted SigNoz Anyway.

Three Tabs Was My Debugging Workflow. Then One Checkout Failed.

Before / after (same broken checkout)

Context: Same Cluster, Existing Stack Left Alone

Install: Helm + NodePort (reproducible)

Sending Real Data: Break Checkout on Purpose

Same-afternoon evidence

What I Touched (Early Win Checklist)

1. Traces — error list for signoz-demo-api

2. Trace detail — the join starts here

3. Logs — same failure, same trace_id

4–5. Metrics + dashboards

6. Alerts — checkout-error-spike

Favorite Feature: The Join I Stopped Building by Hand

What Broke (Tell Past-Me)

What I’d Do Next

Thanks for Reading

Links

Demystifying AWS IAM: Best Practices for User Group Management and Role Creation

1. Traces — error list for `signoz-demo-api`

3. Logs — same failure, same `trace_id`

6. Alerts — `checkout-error-spike`