foresight

I built an agent that writes its own SigNoz dashboards after an incident

The same incident always gets you twice.

It blows up on a Tuesday, you scramble, you patch it. Three weeks later a cousin of that same incident strolls right past your monitoring because nobody went back and added an alert. The fix was "obvious" in hindsight. It just never got written down.

So over a hackathon I built FORESIGHT to kill that gap. It watches live SigNoz telemetry, predicts a breach before it fires, explains the cause in plain English, and then does the part I actually care about: it writes the SigNoz dashboard, alert rule, and runbook for you, so the same thing gets caught next time. Observability that writes itself.

Here's what actually worked, the SigNoz bits I leaned on hard, and a few things that ate an hour each and aren't in any doc.

What it does

Four stages, on a loop:

FORESEE  → poll SigNoz trends, predict an SLO breach before it fires
EXPLAIN  → pull traces/metrics, produce a plain-English root cause
CODIFY   → auto-create a SigNoz dashboard + alert + runbook
LEARN    → measure its own MTTR from alert history, remember the runbook

I needed something realistic to break, so the monitored app is a tiny AI coffee-shop chatbot. It emits OpenTelemetry spans with GenAI attributes, and it's got a "break knob" so I can inject faults live in a demo — a latency ramp, a token-cost ramp, a tool-error spike.

Right after I inject a fault: FORESEE calls the breach, EXPLAIN posts the cause, CODIFY reports the SigNoz stuff it just created — every step with a clickable link.

The GenAI telemetry (and one honest catch)

Each chat request emits one server span, plus a span per tool call. The attribute names follow OpenTelemetry's GenAI semantic conventions. Heads up: they're still experimental. They live under gen_ai.* and aren't stable yet, so pin a version. The ones I used:

• gen_ai.request.model
• gen_ai.usage.input_tokens
• gen_ai.usage.output_tokens
• gen_ai.operation.name
• gen_ai.tool.name (on tool spans, with an OK/ERROR status)

Now the catch that tripped me up: there's no standard gen_ai.usage.cost attribute. The conventions cover the model and token counts. Cost? That's on you. I derive it from the token counts and attach it as a custom attribute — and I don't pretend it's part of the spec. If you're building cost dashboards, that number is yours to own.

with tracer.start_as_current_span("chat", kind=SpanKind.SERVER) as span:
    result = backend.generate(prompt)          # real or simulated LLM
    span.set_attribute("gen_ai.request.model", result.model)
    span.set_attribute("gen_ai.usage.input_tokens", result.input_tokens)
    span.set_attribute("gen_ai.usage.output_tokens", result.output_tokens)
    span.set_attribute("gen_ai.usage.cost", result.cost)   # derived, not standard

One early decision paid off big: the app runs with no API key at all, on a simulated LLM that emits the exact same telemetry shape. I could build and demo the whole pipeline offline, then swap in a real model only when I wanted to.

One /chat request in SigNoz Traces. The `gen_ai. attributes ride along on the span. gen_ai.usage.cost` is the one I added myself.*

Reading and writing SigNoz: MCP first, REST as backup

SigNoz ships an MCP server (Model Context Protocol) that exposes services, traces, dashboards, and alerts as tools. FORESIGHT reaches for MCP first, every read and every write, and only drops to the REST API if MCP's not around. Both go through one error wrapper, so a SigNoz hiccup shows up cleanly instead of taking down the whole loop.

Wiring MCP to a local SigNoz is one JSON block. The gotcha: the auth header is SIGNOZ-API-KEY, hyphenated — not Authorization. I burned a few minutes on 401s before I actually read that line.

{
  "mcpServers": {
    "signoz": {
      "url": "http://localhost:8000/mcp",
      "headers": { "SIGNOZ-API-KEY": "<your-editor-api-key>" }
    }
  }
}

Here's the bit that matters most: reads go through the Query Builder, not raw SQL. Every latency/error/request query is a builder request (queryType: "builder") with an aggregation like p99(duration_nano) and a filter like service.name = '<svc>'. Why bother? Because the same query definition drives both the FORESEE polling and the panels CODIFY generates later. The alert threshold and the dashboard panel end up sharing the exact same query. One source of truth.

The part I care about: CODIFY writes the guards

When an incident resolves, CODIFY builds three things — each on its own:

1. A service-scoped dashboard — p99, error rate, request rate. Plus gen_ai panels if it was an AI incident.
2. A threshold alert rule — "above", target = the SLO, filtered to the service.
3. A runbook — the story, the evidence, the likely cause, and links to the guards it just made.

The decision I'm happiest with: each artifact is attempted on its own. If the alert fails to create, you still get the dashboard and the runbook, and the failure gets logged per-artifact instead of nuking the whole run. The only thing that stops everything is if the failure recorder itself throws — the one case where pushing on would hide a real error.

for artifact, make in (("dashboard", _dashboard), ("alert", _alert), ("runbook", _runbook)):
    try:
        make()                       # each returns an id + a SigNoz link on success
    except Exception as exc:
        record_failure(artifact, exc)  # recorded, loop continues

Nobody hand-built this. CODIFY created the dashboard and a matching p99 alert through the SigNoz API the moment the incident cleared.

Making it trustworthy: property tests caught real bugs

I didn't want a demo that only survives the happy path. So the decision logic — slope prediction, breach math, health classification, the codify failure isolation — is covered by property-based tests with Hypothesis, at least 100 generated cases each. This is where I learned the most, because the generators found bugs I'd never have typed:

• Windows filenames. The runbook writer names files after the service. Hypothesis cheerfully generated a "service name" full of control characters — a perfectly legal SigNoz label, an illegal Windows filename. The write exploded. Real service names are fine, so I constrained the generator to realistic labels. Example-based tests would've sailed right past it.
• A wording mismatch. One test asserted the cause string "token-cost spike" showed up in the signals list. The signal actually read "gen_ai token cost is $X/call." Same meaning, different words — the kind of disagreement only a fuzzer bothers to find.

Two things to know if you pair Hypothesis with pytest fixtures:

• Use a function-scoped fixture inside @given and Hypothesis throws a health check. It's fine when each example re-seeds the fixture — silence it with @settings(suppress_health_check=[HealthCheck.function_scoped_fixture]).
• Run the suite from the project root, not a single file. My pytest.ini sets pythonpath = src, and a few tests import a shared conftest that only resolves on full-directory collection.

Full circle: the agent watches itself

FORESIGHT emits its own OpenTelemetry spans too — one per stage, linked under a single loop trace, each tagged with a foresight.stage attribute and an outcome. So the predict→explain→codify run shows up in SigNoz, right next to the app it's monitoring.

The emitter fails open. If the exporter's down, it logs and moves on. The last thing you want is your observability agent crashing on its own telemetry.

The watcher, watched. FORESIGHT's own investigation is a trace in SigNoz — one span per stage, tagged with foresight.stage.

What I'd tell my past self

• MCP first, REST underneath. MCP kept the code clean; the REST fallback meant a hiccup never killed a demo.
• Cost is your number, not the spec's. Don't sit around waiting for a gen_ai.usage.cost convention that doesn't exist.
• Write the failure-isolation before the happy path. Bolting "keep going if one thing fails" on afterward is way more painful than building it in.
• Let a fuzzer name your files. The property tests earned their keep just by picking inputs I never would.

Wrap-up

Small idea, but it stuck with me: the most valuable thing an incident produces isn't the fix — it's the guard that stops the next one. And that guard almost never gets written. Hand that step to an agent, grounded in the same SigNoz queries a human would run, and it turns out to be both doable and genuinely useful.

Want to try the pieces? The OpenTelemetry GenAI conventions live at opentelemetry.io/docs/specs/semconv/gen-ai, and SigNoz's MCP server and Query Builder are what made the read/write half possible. Start by instrumenting one app with gen_ai.* attributes and watching the traces land. Everything else builds from there.

Checked the OpenTelemetry convention details against the official semantic-conventions docs; the cost caveat reflects that there's no stable gen_ai.usage.cost attribute at the time of writing.