DEV Community: Arjun Sabu

ChronoLens

Arjun Sabu — Sun, 26 Jul 2026 08:19:51 +0000

ChronoLens: proving the outage that never happened

How we built a self-preventing reliability loop on SigNoz — and how we made it prove its own results with measured telemetry instead of a pretty chart.

Built for the Agents of SigNoz hackathon (Track 01 — AI & Agent Observability).
Repo: github.com/greninja-op/ChronoLens

Contents

The problem nobody can demo
Chrono-Proof: a counterfactual made of measurements
Blast-radius: which service falls next
The loop that does the work
Agent Watch: the same loop, pointed at an AI agent
Human-in-the-loop: Slack and WhatsApp
The telemetry stack: OpenTelemetry in, SigNoz out
What SigNoz's APIs taught us
MCP: calling the server, not just resembling it
The demo app and the agent's model
Reproduce it
What we cut, and why it matters
Honest limits

The problem nobody can demo

Observability tells you what broke. Every SRE tool on the market is excellent at the postmortem
and useless at the ten seconds before it. So we built the obvious thing: predict the breach, act
before it lands, verify the fix.

Then we hit the problem that kills every prevention tool.

Prevention is invisible. When it works, nothing happens. There's no incident, no graph spike,
no war room — just a flat line and an engineer with no way to prove the flat line was earned. The
industry's usual answer is a "what would have happened" chart drawn from a formula, and anyone
technical can dismiss it in one sentence: you made that curve up.

We had exactly that in our own codebase. An endpoint called /api/counterfactual that produced a
beautiful dual-timeline chart out of a hardcoded exponential decay, with no telemetry behind it.
We deleted it.

What replaced it is the feature this post is really about.

Chrono-Proof: a counterfactual made of measurements

The honest way to prove a negative is to be strict about which half of the claim is measured and
which half is estimated — and to say so on the artifact itself.

Chrono-Proof does five things:

Pulls the real p99 series for the service out of SigNoz (Query Builder v5, time_series).
Splits that series at the moment ChronoLens acted.
Fits the trend on the pre-action samples only, using the same machinery the forecaster uses — EWMA smoothing, then Holt's linear trend, sanity-blended with least squares so a bad Holt initialisation can't dominate — and extends that slope across the post-action window with a confidence band derived from the residual spread. The slope estimate is careful; the extrapolation itself is deliberately simple and linear, which is why it's labelled an estimate.
Overlays the measured post-action reality from SigNoz.
Quantifies the gap: breach-seconds avoided, peak milliseconds shaved, and SLO-violation area (error budget) saved.

Here is real output from a live run, not a mock-up:

=== CHRONO-PROOF: chronolens-store (source: signoz) ===

  MEASURED (SigNoz)   peak      48 ms ·     0s over SLO · final 45 ms
  PROJECTED (est.)    peak    4474 ms ±1108 ·   90s over SLO · trend +15.2 ms/s

  Breach avoided      90s
  Peak shaved         4427 ms
  Error budget saved  306490 ms·s
  Prevented           True  (confidence 71%)

Every number carries its provenance. The defused arm is measured. The counterfactual arm is a
labelled linear extrapolation with an interval — an estimate, and the note on the payload says
exactly that:

The 'measured' arm is SigNoz data. The 'projected' arm is a linear extrapolation of the measured
pre-action trend (± band) — a labelled estimate, not a measurement.

The bug that made it honest

Our first version guessed when the action happened by taking the series peak. That broke in a
way worth documenting: while a load ramp was still running, p99 climbed again after the fix, the
"peak" landed in the wrong place, and Chrono-Proof reported "the fix did not hold" for a fix
that had held.

The correct answer was already in the system. Every remediation writes a case file to the ledger
with a timestamp, so the proof now derives the action point from the recorded action time:

idx = int(round(n_samples - 1 - (age / step_s)))

It skips non-action rows (none, pre-provision, suggest:*), takes the newest real action, and
falls back to the peak heuristic only when no case matches the window — stating which anchor it
used. Six unit tests cover the arithmetic, the rejections and the fallback.

Blast-radius: which service falls next

Predicting that one service will breach is table stakes. The question that actually matters in a
distributed system is the one nobody answers in advance:

payment is about to go — what goes down with it, in what order, and how long do I have?

SigNoz already knows the topology. It derives a service dependency graph from traces and exposes
it at /api/v1/dependency_graph. We walk that graph upward from the degrading dependency, because
a caller can never be faster than the thing it waits on: a slow child pushes latency into
every ancestor, scaled by how much of the parent's traffic actually depends on that path.

Live output, three real services:

=== BLAST-RADIUS FORECAST (topology: signoz-service-map) ===

  Root cause: chronolens-payments-db  [root hop: payment.db_query]

  service                     p99     slope  inherit   breach in
  chronolens-payments-db *  6824ms    +37.3     +0.0         NOW
  chronolens-payments       6824ms    +37.3    +37.3         NOW
  chronolens-store           951ms    +27.6    +27.6         NOW

Two details that make this real rather than decorative:

The root is the most-downstream degrading service. A service climbing only because its dependency is slow is a symptom. Fixing it mutes the alarm and changes nothing.
Only services inside the dependency graph can be the root. An unconnected sidecar with a steep slope isn't the cause of a cascade — it has no downstream blast path. (We found this by watching it confidently blame our demo AI agent, which nothing depends on.)

One service can't cascade

This feature was untestable in our own demo at first: the store was a single service, so there was
nothing to chain. We split it into a real three-tier topology, each tier emitting spans under its
own service.name with the parent context preserved:

chronolens-store  ──▶  chronolens-payments  ──▶  chronolens-payments-db

Now dependency-slow injected at the deepest tier produces a genuine measured cascade, and SigNoz's
own dependency graph is what ChronoLens reads to trace it.

The loop that does the work

Chrono-Proof and blast-radius are the evidence layer. Underneath is the control loop:

LEARN → FORESEE → CLASSIFY → CASCADE → GOVERN → PREVENT → VERIFY → COOLDOWN → RECORD
  ▲                                                                              │
  └──────────────────── the ledger becomes LEARN's memory ──────────────────────┘

LEARN reads past incidents, including time-of-day seasonality. For a repeat offender it pre-provisions a higher floor before anything degrades, and it corroborates recurrence against SigNoz's own alert state rather than trusting only its local ledger.
FORESEE projects p99 to a time-to-breach behind a confidence guard: enough samples, slope above a noise floor, and a sustained rise. An elevated error rate from a second, independent signal lifts confidence.
CLASSIFY picks the fix that matches the signal, not always "scale": load → scale, dependency → circuit-break, pool → resize, memory → restart, errors → roll back. The errors signal is cross-checked against a SigNoz logs query, so classification spans two signals.
CASCADE names the root hop from a grouped traces query, plus an exemplar trace ID for a deep link.
GOVERN is a trust ladder — suggest (never acts alone), earn (autonomous after N verified saves on that service), auto.
PREVENT acts behind anti-flap guardrails: a minimum dwell time between actions, a hard capacity ceiling, an hourly action budget, and a global kill switch. Every action is reversible and every action stores its precise inverse.
VERIFY asks SigNoz whether the breach was actually avoided. If it wasn't, it rolls back and escalates.
COOLDOWN returns the capacity once load subsides, so prevention isn't paid for with permanent over-provisioning.
RECORD files the receipt that LEARN reads next time.

A run against live SigNoz, abridged:

[FORESEE ] chronolens-store: p99 2540.8ms, rising 8ms/s → SLO breach NOW (confidence 100%)
[CLASSIFY] Signal: load → reversible fix 'scale'
[CASCADE ] Degradation at 'payment.charge' (p99 1005.1ms) (measured in traces) …
           Exemplar trace: a6c3d99cc80d5efc2dfa9f720cdbe773
[GOVERN  ] autonomy=auto — acting automatically
[PREVENT ] Applied 'scale' (reversible). Rollback: scale back down once load subsides
[VERIFY  ] Confirmed via SigNoz: p99 back to 53.5ms — breach avoided
[COOLDOWN] Load subsided — scaled 8.0 → 2.0, returned 6.0 capacity units (~$3.90)
[GUARD   ] Filed a guarding SigNoz alert + dashboard on chronolens-store p99
[RECORD  ] Case filed: breach avoided · returned 6.0 units (~$3.90)

Agent Watch: the same loop, pointed at an AI agent

Track 01 is agent observability, so ChronoLens watches an agent the way it watches a service — and
agents fail in ways HTTP status codes never capture.

The demo agent is a café assistant instrumented with OpenTelemetry GenAI semantic-convention
attributes. Each turn emits an agent.turn parent span with child gen_ai.chat and
tool.execute spans, carrying gen_ai.request.model, gen_ai.usage.input_tokens /
output_tokens, llm.step_count, llm.cost_usd and agent.tools. It runs in three modes —
normal (the learned baseline), drift, and loop — so each failure is reproducible on demand.

Three analyzers read those spans:

Analyzer	The failure it catches
Behaviour drift	After a prompt tweak or model swap the agent still returns 200 OK with normal latency, but it now calls a tool it never used, takes more steps, or writes far longer answers. A fingerprint (tool distribution, model mix, avg steps, avg tokens) is scored against a saved baseline.
Loop / cost breaker	The agent reasons in circles, calling the same tool repeatedly, burning tokens with no crash. It fires on no progress and on a cost budget, not just a clock — so a long but genuinely productive turn is left alone.
Answer quality	Grades recent answers to separate changed from worse — drift is a change signal, not a verdict. The answers are read from SigNoz logs: the agent emits each full response as an OTel log record, because span attributes only carry a truncated preview and you cannot grade what you cannot read.

Detecting from SigNoz, not by poking the agent

This is the part we had to fix to be honest. The analyzers originally called the agent's /chat
endpoint to get turns — which means "agent observability" that never touched the observability
platform. Now the default path issues a raw traces query (requestType: "raw") against SigNoz
for the agent's agent.turn spans and reconstructs the turns from their attributes:

turns read FROM SIGNOZ: 16
   gpt-4o-mini  2 steps  ['get_menu', 'place_order']  $9e-05  src=signoz

Every response now reports data_source: "signoz", and falls back to driving the agent only when
SigNoz has no spans yet (a cold stack) — saying so when it does.

Quality grading was the last hold-out, and fixing it needed a second signal rather than a cleverer
query: the agent now emits its complete response as an OTel log record, and the judge reads those
bodies back with a requestType:"raw" logs query. So the drift and loop analyzers read traces,
the judge reads logs, and all three are telemetry-driven — verified live on 8 graded answers.

Human-in-the-loop: Slack and WhatsApp

The trust ladder's suggest tier is only meaningful if a human can actually approve something. So
when GOVERN decides ChronoLens may not act alone, it posts an interactive approval card:

🕳️ ChronoLens needs your approval
Service: payment · Forecast: p99 past SLO in ~85s (confidence 86%)
Signal: pool · Proposed fix: pool-resize — reversible
[ ✅ Approve ] [ ✋ Deny ]

Tapping Approve runs the real PREVENT → VERIFY → COOLDOWN → RECORD path and then edits the
same message with the SigNoz-verified outcome ("p99 back to 53.5 ms — breach avoided"). Deny
records the decision and stands down. Agent anomalies get their own card with
Break / pin baseline, which pins the agent to its last-good baseline and verifies the next turn.

Every tap answers twice. Remediation is a real verify cycle, so it takes tens of seconds — and a
button that goes quiet for half a minute reads as a button that didn't work, so people tap it again.
Each tap now gets an immediate acknowledgement ("⏳ Applying scale_out…") and then the verified
outcome once SigNoz has confirmed it. On Slack that's one message rewritten twice; on WhatsApp it's
two replies. Unrecognised taps get an answer too — silence is indistinguishable from a broken
integration.

Both surfaces call the same approval engine, tagged with the surface it came from, so an approval
means exactly the same thing wherever it arrives and the ledger receipt records which surface
decided. WhatsApp previously re-ran the whole loop instead of executing the approved action, which
meant a tap on your phone and a tap in Slack didn't do quite the same thing. One engine, two doors.

Do you need both? No — and we're explicit about it:

Channel	Mechanism	When to use it
Slack	Socket Mode (an outbound WebSocket), so no public URL is needed. Free tier is enough: a bot token (`xoxb-`) plus an app-level token (`xapp-`).	The default. Least friction, richest UI (Block Kit buttons), ideal for a team channel and for local development.
WhatsApp	Meta WhatsApp Cloud API webhooks with HMAC-SHA256 signature verification, interactive reply buttons. Needs a public HTTPS endpoint and a Meta business number.	Reach — an on-call engineer who isn't at a desk. Convenience, not capability.

Either one alone is sufficient. They implement the same contract; Slack is the recommended path and
WhatsApp exists because approving a production fix from a phone lock screen is genuinely useful.
Everything shown in this post is Slack, deliberately — we'd rather describe the WhatsApp path than
screenshot one we aren't actively driving.

A note on the signature check

Our first implementation of the WhatsApp webhook returned True when the app secret or signature
header was missing — a development shortcut that accepts unsigned requests. It now fails closed
whenever a secret is configured: a missing signature is rejected. The only permissive case is local
development with no secret set at all, and the README says not to expose the webhook without one.

Worth flagging because it's the kind of shortcut that quietly ships.

The telemetry stack: OpenTelemetry in, SigNoz out

Nothing in ChronoLens has a private data path. Everything it knows arrives as OpenTelemetry and
everything it concludes is written back to SigNoz.

What we emit (OpenTelemetry):

The demo services export OTLP traces over gRPC (:4317) with service.name per service, so SigNoz derives the dependency graph itself rather than being told about it.
The demo agent adds GenAI semantic-convention attributes — gen_ai.request.model, gen_ai.usage.input_tokens / output_tokens, plus llm.step_count, llm.cost_usd, agent.tools — on agent.turn / gen_ai.chat / tool.execute spans.
ChronoLens instruments itself: each loop stage is a chronolens.stage span under one loop trace, and it exports its own metrics (chronolens.prevented_total, cost_saved_usd, seconds_to_breach). Its guard dashboard then reads those metrics back out of SigNoz — the loop watching the loop.

What we read (SigNoz):

Surface	Used for
Query Builder v5 — traces (`scalar`, `time_series`, `raw`, `group_by`)	p99 per service, p99 series for forecasting and Chrono-Proof, per-span-name breakdown for the cascade root, and raw `agent.turn` spans for Agent Watch
Query Builder v5 — logs	`severity_text='ERROR'` counts that cross-check the `errors` classification against a second signal
Metrics read-back	ChronoLens's own gauges, so the dashboard closes the circle
Service dependency graph	The real topology behind the blast-radius forecast
Exemplar trace IDs	Deep links from a receipt into the exact trace
Alert state / history	LEARN confirms recurrence from SigNoz's firing rules, not just its local ledger

What we write (SigNoz):

Surface	Used for
Threshold alert rules	A guard alert per prevented incident, so it stays watched
Dashboards	A guard dashboard with a p99 panel and a panel reading ChronoLens's own metric
Saved views	A pinned Traces-explorer view for the guarded service
Silences	Muted while the loop actively remediates, lifted after VERIFY — nobody is paged for a fix already in flight
Notification channels	Discovered and reused, so ChronoLens routes its notes through the same channel an alert would
GenAI guard dashboard	Auto-filed for the watched agent: cost per turn, steps vs ceiling, output tokens, tool-call mix, and turn latency for contrast
Anomaly alert rule	A learned baseline on agent cost per turn, not a fixed threshold — catches "abnormal for this hour" while still inside the budget

What SigNoz's APIs taught us

Writing to an observability platform is a different skill from reading from one, and four of these
cost us real debugging time. None of them produce an error message that names the cause.

Anomaly rules only accept METRIC_BASED_ALERT. Pointing one at a traces query —
p99(duration_nano) grouped by service — is rejected outright:
anomaly_rule can only be used with METRIC_BASED_ALERT. So "alert when this service's
latency is weird for a Tuesday" isn't directly expressible on spans. The fix was to make
the agent emit its per-turn shape as metrics (chronolens.agent.cost_usd,
.steps, .output_tokens) alongside its spans, and anomaly-alert on the metric. That's
a better design anyway: traces answer "what happened in this turn", a metric gives the
continuous series a seasonal baseline can be learned from.

The v2 REST endpoint rejects the v1 rule schema silently. Anomaly rules use the older
v1 shape (top-level evalWindow/frequency, condition.op/matchType/target/
algorithm/seasonality) and our POST /api/v2/rules returned
{"message":"validation failed","errors":[]} — an empty error list, no field named.
Rather than keep guessing, we filed it through the MCP server's signoz_create_alert
tool, which owns that version handling. It worked first time, and it upgraded our MCP
usage from read-only to read and write.

The anomaly rule went to state: firing within a minute of creation — it had already
noticed the cost-per-turn spike from the loop-mode demo.

A dashboard can store every panel and still render blank. Our first auto-filed dashboards
looked perfect over the API — five widgets, correct queries — and the UI showed "Welcome to your new
dashboard." Nothing errored. SigNoz positions panels from a layout array (a react-grid spec keyed
by widget id) and we had sent neither layout nor widget ids, so there was nothing to place. The
same class of failure hides in every field the front-end maps over but the API doesn't require:
builder.queryFormulas, promql, clickhouse_sql, selectedLogFields, selectedTracesFields,
contextLinks.linksData, and the dashboard's own variables. Missing means undefined, not empty.
We now send empties for all of them and a test asserts every panel carries them, so an empty
dashboard can't ship again.

Threshold markers need the long field names. {"index","label","value","unit"} is accepted and
silently draws nothing; the UI reads thresholdValue, thresholdUnit, thresholdOperator,
thresholdFormat and thresholdColor. And latency thresholds are nanoseconds on dashboards but
milliseconds on alerts — set the dashboard marker in ms and your SLO line lands in the wrong place
by six orders of magnitude.

A metric only emitted during a run looks identical to a broken metric. Our "incidents prevented"
panel read No Data over a 30-minute window even though the ledger had saves in it: the gauge was
only published while a loop was executing, so the newest sample was hours old. Mission Control now
publishes it on a heartbeat, which turns one dot per incident into a continuous line. Worth
generalising — if you write your own metrics back to your observability platform, emit them on a
timer, not only on the event.

MCP: calling the server, not just resembling it

Foundry installs the SigNoz MCP server alongside SigNoz, and it's tempting to call your queries
"MCP-compatible" because they have the same shape. We did exactly that for a while — then admitted it
was a claim about resemblance, not usage, and wrote a real client.

src/chronolens/mcp.py is a dependency-free JSON-RPC client: it performs the initialize handshake,
sends notifications/initialized, lists the server's tools, and invokes them with tools/call.
Against the live server it reports SigNozMCP, protocol 2024-11-05, and 41 tools.

Four things the protocol taught us, none of them in our first guess:

Auth is mandatory — no SIGNOZ-API-KEY header gets you 401 Authorization or SIGNOZ-API-KEY header required.
The Accept header must allow both application/json and text/event-stream, or the Streamable-HTTP transport refuses the request.
A reply may arrive as an SSE frame (data: {...}) instead of a JSON body, so the parser handles both.
Tool results are JSON nested inside a text block (result.content[].text) — a second decode.

The co-pilot then routes a plain-English question to real tool calls, and the UI lists every call with
its arguments and row count, so the answer is auditable rather than asserted:

Q: which services are slowest right now?      → signoz_list_services            (5 rows)
Q: are any alerts firing?                     → signoz_list_alert_rules         (10 rows)
Q: any error logs in the last hour?           → signoz_search_logs              (1 row)
Q: top operations for chronolens-store        → signoz_get_service_top_operations (4 rows)

Routing is rule-based on purpose: intent here is a small closed set, an LLM would add latency and a
failure mode for no accuracy gain, and a reviewer can read the table and check it. We also removed an
LLM "phrasing" pass that had been rewriting correct answers into generic remediation prose — it turned
a true answer into a plausible-sounding wrong one, which is the exact failure this project exists to catch.

The reads above still go through the REST Query Builder on the hot path, because the control loop
needs tight latency and deterministic shapes; MCP is how ChronoLens answers questions, and both
paths hit the same SigNoz.

The demo app and the agent's model

Two things are watched, and it matters that we're precise about what's real.

The demo store is a synthetic three-tier app (chronolens-store → chronolens-payments →
chronolens-payments-db, each its own service.name) with two jobs: emit believable traces
SigNoz can derive a dependency graph from, and expose reversible levers — scale,
pool-resize, circuit-break, restart, roll back, reset. Its faults ramp gradually
(traffic-ramp, dependency-slow, pool-leak, memory-leak, error-spike) because a step
function isn't forecastable and a forecast you can't test isn't a forecast.

The agent's model is simulated by default, and we say so. The café assistant's three modes
(normal / drift / loop) produce deterministic token counts, tool sequences and costs from a
price table. That's a deliberate choice: a drift demo has to be reproducible, and a real model
that happens to answer consistently proves nothing. The spans it emits are real OpenTelemetry
GenAI spans either way, which is what the analyzers consume.

For real inference, LLM_PROVIDER switches the explanation layer and the agent to a live backend —
openai, bedrock (AWS), azure, or gemini — via one API key. With no key set, ChronoLens runs
end-to-end on a rule-based explainer. Nothing in the loop requires an LLM to function; the LLM only
makes the narration nicer, which is the right place for a non-deterministic component.

Reproduce it

The hackathon asks for a reproducible install, so the repo ships casting.yaml +
casting.yaml.lock and Foundry brings up SigNoz and its MCP server in one command:

bash scripts/bringup.sh          # SigNoz UI :8080 · OTLP :4317/:4318 · MCP :8000/mcp
cp .env.example .env             # add SIGNOZ_API_KEY
pip install -r requirements.txt
python -m demo_store.store       # the watched services  :8090
python demo_agent/agent.py       # the watched agent      :8091
python app.py                    # Mission Control        :8095

Then drive it from the UI, or:

python -m chronolens.cli respond   # one full loop
python -m chronolens.cli proof     # the measured counterfactual
python -m chronolens.cli blast     # who falls next
python -m chronolens.cli guard     # file the agent dashboard + alerts (incl. the anomaly rule via MCP)
python -m chronolens.cli slack     # the approval listener

python scripts/demo_check.py runs 23 end-to-end checks (stack up, services reporting into SigNoz,
each API surface answering) and prints what's missing rather than failing obscurely. Both demo
services are self-driving — the store at ~6 rps, the agent at ~0.5 turns/s — so a fresh clone shows
populated dashboards instead of an empty SigNoz, which is a failure mode we hit ourselves.

If you'd rather not run anything, both dashboards are committed as importable JSON in
dashboards/: in SigNoz go Dashboards → + New dashboard → Import JSON and paste the file. They
are generated from the same code that files them over the API (scripts/export_dashboards.py), so
they can't drift from what the project actually creates.

The test suite is 179 tests — unit plus property-based with Hypothesis — and most of the
interesting ones are regressions for bugs found against live telemetry rather than in CI: the
negative projection, the noise-level slope that hijacked the cascade root, the blank dashboard, the
approval that answered too late.

What we cut, and why it matters

Late on, we audited our own repo and deleted six features: a synthetic "counterfactual" chart that
drew a curve from a formula, a "stress test" that ran a hardcoded array and claimed to read SigNoz,
a tool circuit-breaker rebuilt on every request so it forgot instantly, an executive ROI report that
asserted "100% verification rate" as a string literal, a translation layer unrelated to
observability, and a token throttle nothing read.

They demoed fine. That's the problem — each one was a place where a judge opening the file would
find a claim the code didn't support, which poisons trust in the features that are real. Two
docstrings claimed "real-time OpenTelemetry span metrics from SigNoz" in modules containing no
SigNoz call at all.

The same pass fixed a webhook that accepted unsigned requests, an endpoint that returned its own
verify token, and a real-looking phone number committed as a default in source.

Fewer features, each of which survives being read. That's the trade we'd make again.

Honest limits

The projected arm of Chrono-Proof is a linear extrapolation of a measured trend, not a measurement. Real systems plateau under saturation, so a long projection is an upper bound. Every field is labelled by provenance and the confidence score falls as the pre-action trend gets noisier.
WhatsApp needs a public callback URL to close its loop. Outbound cards send without one, but the button tap is a Meta webhook, and Meta access tokens expire — so Slack (Socket Mode, outbound only) is the path we demo and recommend. The code is surface-agnostic; the operational burden is not.
All three Agent Watch analyzers now read SigNoz. Grading was the last hold-out: span attributes carry only a truncated preview of the response, so the agent ships each full answer as an OTel log record and the judge reads it back with a requestType:"raw" logs query (data_source: "signoz", verified on 8 graded answers). If the logs pipeline is cold it falls back to driving the agent and labels itself agent-driven rather than implying telemetry.
The agent is simulated by default (see above).
The blast radius is only as good as the dependency graph — with a single service there is nothing to chain, and it says topology_source: unavailable rather than inventing edges.
The AWS serverless template is a scaffold that has never been deployed. It's a complete, reviewable SAM stack (Lambda + EventBridge + DynamoDB + Bedrock), and infra/README.md says plainly that it's unproven — ChronoLens was built and demoed entirely against a local Foundry-installed SigNoz, with a rule-based explainer that needs no cloud.

What we'd build next

Three things, in order of how much they'd change the product:

Rank the blast radius by business impact, not time-to-breach. "Checkout falls in 40s" and "the recommendations sidecar falls in 40s" are not the same incident, and the ordering should know that.
Let LEARN tune its own confidence guard. The noise floor and minimum-samples thresholds are configured today; the ledger already records every false start, so the guard could calibrate itself against its own history instead of a constant we picked.
Deploy the AWS scaffold and prove it. It's a reviewable SAM stack today and nothing more, which is why the blog says so rather than implying otherwise.

Everything else we'd change is plumbing. The idea we keep coming back to is the one this project
started from: an outage that never happened should still leave evidence.

ChronoLens was built for the Agents of SigNoz hackathon, Track 1 — AI & Agent Observability.
Code, casting.yaml and the importable dashboards:
github.com/greninja-op/ChronoLens.

AI assistants were used during this build — an agentic IDE for implementation, tests and drafting,
including driving the SigNoz MCP server to create and inspect dashboards and alert rules. The
architecture, the honesty rules the project is built on, every acceptance decision and everything
that got cut were mine. Each claim in this post was verified against a live SigNoz deployment.

ChronoLens

Arjun Sabu — Sun, 26 Jul 2026 08:02:46 +0000

DDSDSD

foresight

Arjun Sabu — Sat, 18 Jul 2026 18:04:25 +0000

I built an agent that writes its own SigNoz dashboards after an incident

The same incident always gets you twice.

It blows up on a Tuesday, you scramble, you patch it. Three weeks later a cousin of that same incident strolls right past your monitoring because nobody went back and added an alert. The fix was "obvious" in hindsight. It just never got written down.

So over a hackathon I built FORESIGHT to kill that gap. It watches live SigNoz telemetry, predicts a breach before it fires, explains the cause in plain English, and then does the part I actually care about: it writes the SigNoz dashboard, alert rule, and runbook for you, so the same thing gets caught next time. Observability that writes itself.

Here's what actually worked, the SigNoz bits I leaned on hard, and a few things that ate an hour each and aren't in any doc.

What it does

Four stages, on a loop:

FORESEE  → poll SigNoz trends, predict an SLO breach before it fires
EXPLAIN  → pull traces/metrics, produce a plain-English root cause
CODIFY   → auto-create a SigNoz dashboard + alert + runbook
LEARN    → measure its own MTTR from alert history, remember the runbook

I needed something realistic to break, so the monitored app is a tiny AI coffee-shop chatbot. It emits OpenTelemetry spans with GenAI attributes, and it's got a "break knob" so I can inject faults live in a demo — a latency ramp, a token-cost ramp, a tool-error spike.

Right after I inject a fault: FORESEE calls the breach, EXPLAIN posts the cause, CODIFY reports the SigNoz stuff it just created — every step with a clickable link.

The GenAI telemetry (and one honest catch)

Each chat request emits one server span, plus a span per tool call. The attribute names follow OpenTelemetry's GenAI semantic conventions. Heads up: they're still experimental. They live under gen_ai.* and aren't stable yet, so pin a version. The ones I used:

gen_ai.request.model
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
gen_ai.operation.name
gen_ai.tool.name (on tool spans, with an OK/ERROR status)

Now the catch that tripped me up: there's no standard gen_ai.usage.cost attribute. The conventions cover the model and token counts. Cost? That's on you. I derive it from the token counts and attach it as a custom attribute — and I don't pretend it's part of the spec. If you're building cost dashboards, that number is yours to own.

with tracer.start_as_current_span("chat", kind=SpanKind.SERVER) as span:
    result = backend.generate(prompt)          # real or simulated LLM
    span.set_attribute("gen_ai.request.model", result.model)
    span.set_attribute("gen_ai.usage.input_tokens", result.input_tokens)
    span.set_attribute("gen_ai.usage.output_tokens", result.output_tokens)
    span.set_attribute("gen_ai.usage.cost", result.cost)   # derived, not standard

One early decision paid off big: the app runs with no API key at all, on a simulated LLM that emits the exact same telemetry shape. I could build and demo the whole pipeline offline, then swap in a real model only when I wanted to.

One /chat request in SigNoz Traces. The `gen_ai. attributes ride along on the span. gen_ai.usage.cost` is the one I added myself.*

Reading and writing SigNoz: MCP first, REST as backup

SigNoz ships an MCP server (Model Context Protocol) that exposes services, traces, dashboards, and alerts as tools. FORESIGHT reaches for MCP first, every read and every write, and only drops to the REST API if MCP's not around. Both go through one error wrapper, so a SigNoz hiccup shows up cleanly instead of taking down the whole loop.

Wiring MCP to a local SigNoz is one JSON block. The gotcha: the auth header is SIGNOZ-API-KEY, hyphenated — not Authorization. I burned a few minutes on 401s before I actually read that line.

{
  "mcpServers": {
    "signoz": {
      "url": "http://localhost:8000/mcp",
      "headers": { "SIGNOZ-API-KEY": "<your-editor-api-key>" }
    }
  }
}

Here's the bit that matters most: reads go through the Query Builder, not raw SQL. Every latency/error/request query is a builder request (queryType: "builder") with an aggregation like p99(duration_nano) and a filter like service.name = '<svc>'. Why bother? Because the same query definition drives both the FORESEE polling and the panels CODIFY generates later. The alert threshold and the dashboard panel end up sharing the exact same query. One source of truth.

The part I care about: CODIFY writes the guards

When an incident resolves, CODIFY builds three things — each on its own:

A service-scoped dashboard — p99, error rate, request rate. Plus gen_ai panels if it was an AI incident.
A threshold alert rule — "above", target = the SLO, filtered to the service.
A runbook — the story, the evidence, the likely cause, and links to the guards it just made.

The decision I'm happiest with: each artifact is attempted on its own. If the alert fails to create, you still get the dashboard and the runbook, and the failure gets logged per-artifact instead of nuking the whole run. The only thing that stops everything is if the failure recorder itself throws — the one case where pushing on would hide a real error.

for artifact, make in (("dashboard", _dashboard), ("alert", _alert), ("runbook", _runbook)):
    try:
        make()                       # each returns an id + a SigNoz link on success
    except Exception as exc:
        record_failure(artifact, exc)  # recorded, loop continues

Nobody hand-built this. CODIFY created the dashboard and a matching p99 alert through the SigNoz API the moment the incident cleared.

Making it trustworthy: property tests caught real bugs

I didn't want a demo that only survives the happy path. So the decision logic — slope prediction, breach math, health classification, the codify failure isolation — is covered by property-based tests with Hypothesis, at least 100 generated cases each. This is where I learned the most, because the generators found bugs I'd never have typed:

Windows filenames. The runbook writer names files after the service. Hypothesis cheerfully generated a "service name" full of control characters — a perfectly legal SigNoz label, an illegal Windows filename. The write exploded. Real service names are fine, so I constrained the generator to realistic labels. Example-based tests would've sailed right past it.
A wording mismatch. One test asserted the cause string "token-cost spike" showed up in the signals list. The signal actually read "gen_ai token cost is $X/call." Same meaning, different words — the kind of disagreement only a fuzzer bothers to find.

Two things to know if you pair Hypothesis with pytest fixtures:

Use a function-scoped fixture inside @given and Hypothesis throws a health check. It's fine when each example re-seeds the fixture — silence it with @settings(suppress_health_check=[HealthCheck.function_scoped_fixture]).
Run the suite from the project root, not a single file. My pytest.ini sets pythonpath = src, and a few tests import a shared conftest that only resolves on full-directory collection.

Full circle: the agent watches itself

FORESIGHT emits its own OpenTelemetry spans too — one per stage, linked under a single loop trace, each tagged with a foresight.stage attribute and an outcome. So the predict→explain→codify run shows up in SigNoz, right next to the app it's monitoring.

The emitter fails open. If the exporter's down, it logs and moves on. The last thing you want is your observability agent crashing on its own telemetry.

The watcher, watched. FORESIGHT's own investigation is a trace in SigNoz — one span per stage, tagged with foresight.stage.

What I'd tell my past self

MCP first, REST underneath. MCP kept the code clean; the REST fallback meant a hiccup never killed a demo.
Cost is your number, not the spec's. Don't sit around waiting for a gen_ai.usage.cost convention that doesn't exist.
Write the failure-isolation before the happy path. Bolting "keep going if one thing fails" on afterward is way more painful than building it in.
Let a fuzzer name your files. The property tests earned their keep just by picking inputs I never would.

Wrap-up

Small idea, but it stuck with me: the most valuable thing an incident produces isn't the fix — it's the guard that stops the next one. And that guard almost never gets written. Hand that step to an agent, grounded in the same SigNoz queries a human would run, and it turns out to be both doable and genuinely useful.

Want to try the pieces? The OpenTelemetry GenAI conventions live at opentelemetry.io/docs/specs/semconv/gen-ai, and SigNoz's MCP server and Query Builder are what made the read/write half possible. Start by instrumenting one app with gen_ai.* attributes and watching the traces land. Everything else builds from there.

Checked the OpenTelemetry convention details against the official semantic-conventions docs; the cost caveat reflects that there's no stable gen_ai.usage.cost attribute at the time of writing.