One governed loop, three SigNoz tracks: an agent that heals, prices its own carbon, and gates accessibility

Every "AI plus observability" demo I had seen stopped in the same place. A
beautiful trace of the thing going wrong. The dashboard lights up, the tokens are
gone, and then a human reads it and fixes it by hand.

Observability tells you the agent is on fire. It does not put the fire out.

So for the Agents of SigNoz hackathon I built observable-agent, a local model on
my own laptop wired to self hosted SigNoz, and I set myself one rule. Do not just
watch the agent. Act on its own telemetry, safely, and prove the fix in telemetry
afterwards. Then I pushed that single idea across all three tracks, because the
same primitives that heal a reliability breach also price its carbon and gate its
accessibility.

Here is the whole project, and exactly how SigNoz holds it together.

The primitives, and the foundation they sit on

The design rests on three moves that repeat in every track:

1. Record raw facts as OpenTelemetry, never a summary.
2. Grade those facts with a fail closed, three state verdict (PASS, BREACH, or UNKNOWN, never a comforting false zero).
3. Alert on the verdict in SigNoz, and where it is safe, act on it.

The foundation is deliberately boring and reproducible. SigNoz v0.133 runs on my
laptop, stood up with Foundry (WSL2 plus Docker), and the repo ships the exact
casting.yaml and casting.yaml.lock so a judge can re-run the same deployment.
The agent runs on a local Ollama model (qwen2.5:3b), no cloud and no API keys.
Every request emits an agent.invoke to llm.chat to tool.* span tree using the
OpenTelemetry GenAI semantic conventions, with tokens, cost, and latency dual
written to metrics.

In the loop, SigNoz plays four roles, and because all four read the same
telemetry, they share one source of truth: sensor, diagnostic surface (through the
SigNoz MCP server), scoreboard, and trigger.

Track 01: the self healing SRE sidekick

The first breach I chased is the retry tax. A fault drops the first completed LLM
response, so the agent runs the inference again and spends the tokens twice. It is
a perfect target: it is real telemetry (a duplicate llm.chat span tagged
retry.reason = "response_dropped"), it is measurable as an SLO (dropped divided
by total per cohort), and there is more than one valid fix, so the model has a
genuine decision to make.

One command runs the whole thing, and the entire cycle is a single distributed
trace.

Detection and verification are deterministic signoz_aggregate_traces queries. No
LLM is involved in judging. The model is only asked to decide what to do about a
confirmed breach. Its first move is a read_incident tool call, backed by two live
queries over the MCP server, and the structured evidence it gets back (a retry
rate, a dropped_and_retried count, and a root cause) comes straight out of the
traces the workload had just emitted.

Given that evidence, the local model chose disable_fault_injection on its own and
called it. Retry rate went from 40 percent to 0 percent, and every number in that
sentence is a SigNoz query, not a print statement.

A real alert is the trigger, not a cron job

The hands off path does not poll on a timer. A real SigNoz alert rule fires the
heal. When the retry rate crosses the SLO, SigNoz flips the rule to firing, a
bridge catches it over an Alertmanager webhook, opens a heal.trigger span, and
launches the governed loop. The same alert flipping back to resolved is how the
loop knows it is done. The alert and the heal are one distributed trace.

The part that makes it safe: a policy gate

Letting software act on production is only reasonable if the action is
constrained, and this is the bit most autonomous agent demos skip. Every actuator
mutation routes through a policy gate first. Autonomy runs on four levels (observe,
suggest, approve, auto), and each action carries a risk, a reversibility, and a
blast radius. In auto with a low risk cap, a low risk reversible action applies
itself, while a riskier action like switch_model is held for a human. Every
decision writes an audit line as a heal.policy metric, and if an applied fix does
not clear the breach, the loop restores a snapshot it took beforehand.

The second incident shows the gate earning its keep. A runaway agent stuck in a
loop runs up an unbounded bill. The sensor measures calls per request from the
traces against an SLO ceiling of 6, the model reads the runaway from SigNoz and
arms a set_cost_budget circuit breaker, and here is my favourite frame of the
whole project: the entire policy decision recorded on the span.

That breaker is a structural cut, not a warning. When a request crosses the budget
the agent stops issuing LLM calls and tags the span agent.request.severed = true.
Spend per request dropped from 0.000700 dollars to 0.000122 dollars, about an 82
percent cut, which the plug and play economics layer prices at roughly 1,734
dollars a month at 100,000 requests a day.

One more thing the loop learns: the fix is written to a verified memory in SigNoz,
so the next time the same incident recurs it heals from memory with no model call
at all, which is both faster and calibration free. Running it again while writing
this, a fresh 33 percent retry breach healed straight from memory (proven four
times over now), no model call, MTTR 149 seconds, trace
d0c14392999aef86af95032a8dd080e0.

Track 02: an MCP Contract Lab, three signals, and a zero code proxy

Track 02 is about instrumentation and dashboards, so I aimed the same primitives at
the Model Context Protocol itself.

mcp2_cert is an active probe that certifies an MCP server against a contract. It
exercises initialize, tools/list, and tools/call, grades each result with the
same three state verdict, and emits every exchange as an mcp.* span with a stable
fingerprint. Against my SigNoz MCP server it returns CERTIFIED with fingerprint
ca5859fcf24f5e03.

Then I built the passive companion: an always on instrumentation proxy
(mcp-proxy). It sits transparently between any MCP client and server and turns
their JSON RPC traffic into mcp.<method> spans and mcp.client.* metrics with
zero code changes to either side. To prove it is faithful, I drove the full
certification battery through the proxy. The result was still CERTIFIED with the
identical fingerprint ca5859fcf24f5e03, while the proxy independently emitted
correctly graded spans for every call. You can instrument an MCP integration you do
not own, and lose nothing.

All of that lands on a three signal dashboard (traces, metrics, and logs together)
built through the SigNoz v5 Query Builder API. Every panel self verifies against
the query_range API before the dashboard is created, so a panel that cannot
resolve never ships.

Track 03: energy, carbon, and accessibility as telemetry

Track 03 says observe anything, so I observed two things nobody usually treats as
telemetry.

WattTrace GreenOps measures joules and grams of CO2e per verified answer. It prices
every token to energy on a deterministic token basis, folds a run to its weakest
provenance (so a fallback estimate is never shown as a hardware reading), and
grades it fail closed against a budget. Running a control cohort against a retry
fault, the same verified answers cost 753 J against 1101 J, a 46 percent energy
regression, most of it pure waste on the retried calls. The retry tax, reframed as
a sustainability regression you can alert on.

AccessTrace applies the exact same fail closed primitives to web accessibility. A
real headless browser walks a WCAG journey (page load, navigation, main content,
signup form), runs axe-core scoped at each stage, and emits the journey as a trace:
access.suite to access.journey to access.step. The killer view is that the
access.step spans localise which part of the user flow breaks. On a broken demo
page the verdict is BREACH with a weighted debt of 40 (two critical and four
serious issues, at the exact failing stage), and its fixed twin scores 0. That is a
100 percent reduction, and two live SigNoz alerts fire on the breach.

The cross track finale: carbon becomes a heal sensor

This is where the three tracks stop being three projects. The WattTrace energy
verdict from Track 03 becomes a heal sensor for the Track 01 self healer, read over
the Track 02 MCP protocol. The self healer detects a carbon SLO breach, defined as
the share of a cohort's inference energy wasted on dropped and retried calls, and
closes it through the same governed loop.

The calibration behind that is the interesting part. A healthy multi call workload
sits around 867 J per answer, only about 4 percent under the 900 J single call
reference budget, so gating the verify step on the absolute cap would roll back a
perfectly good fix on ordinary token noise. Gating on wasted fraction instead is
calibration free: a healthy cohort wastes zero, so a verified fix is never rolled
back. In a live carbon heal the footprint per answer dropped from 0.3275 gCO2e to
0.1183 gCO2e (2649 J to 958 J), and the fix was learned to memory. One loop, three
tracks, one trace.

How I used SigNoz, concretely

• Traces: the full agent.heal, watt.suite, access.suite, and mcp.* span trees, every step correlated under one trace id.
• Metrics: heal.slo.breach, heal.mttr, heal.policy, watttrace.energy.consumed, watttrace.carbon.emitted, and more, dual written alongside the traces.
• Logs: trace correlated, structured heal lifecycle logs, so the loop shows up in all three signals.
• Query Builder v5: every detector and every dashboard panel is a programmatic SigNoz query_range query. Detection, diagnosis, and verification are all just SigNoz queries, so the loop is built on the query surface, not merely pointed at it.
• Dashboards: five importable dashboards (self healing, three signal reliability, MCP Contract Lab, WattTrace GreenOps, and AccessTrace), each self verified before creation, with their uuids checked into the repo.
• Alerts: ten live rules across the tracks, and a real alert is the trigger for the heal, not just a notification.
• The SigNoz MCP server: the model reads its own incidents through it, and the proxy instruments it.
• Service accounts and API keys for programmatic access, plus the Foundry casting.yaml and casting.yaml.lock so the whole thing reproduces.

The honest bits

The hackathon asks for real experience, so here is what actually bit me. Small
models need a firm hand to tool call: llama3.2:3b kept emitting a malformed tool
call, so I moved to qwen2.5:3b, and even then the prompt has to spell out that
read_incident only reads and must be followed by a remediation. Keep the LLM out
of the reliability hot path: detection and verification stay deterministic, the
model only decides, and SigNoz judges. The observer effect is real, so the healer
and the workload run as separate services and every query is scoped to a single
cohort tag. CPU inference is genuinely slow, 10 to 110 seconds per call, which
means the latency tail is real and MTTR actually means something. And everything is
fail closed: a run that cannot be judged is UNKNOWN, never a false PASS, because a
zero energy reading is a broken measurement, not free work.

Reproduce it

It is one command per incident, the repo ships the exact casting.yaml so you can
stand up the same SigNoz deployment, and 234 network free unit tests cover the pure
grading logic offline:
github.com/gajanangitte/observable-agent.

Observability that acts, across reliability, protocol, carbon, and accessibility.
It turns out the hard part was never the acting. It was proving, in telemetry, that
the fire is out, the bill is capped, the carbon is cut, and the front door is open
to everyone.