DEV Community: Gajanan Gitte

One governed loop, three SigNoz tracks: an agent that heals, prices its own carbon, and gates accessibility

Gajanan Gitte — Sat, 25 Jul 2026 13:53:23 +0000

Every "AI plus observability" demo I had seen stopped in the same place. A
beautiful trace of the thing going wrong. The dashboard lights up, the tokens are
gone, and then a human reads it and fixes it by hand.

Observability tells you the agent is on fire. It does not put the fire out.

So for the Agents of SigNoz hackathon I built observable-agent, a local model on
my own laptop wired to self hosted SigNoz, and I set myself one rule. Do not just
watch the agent. Act on its own telemetry, safely, and prove the fix in telemetry
afterwards. Then I pushed that single idea across all three tracks, because the
same primitives that heal a reliability breach also price its carbon and gate its
accessibility.

Here is the whole project, and exactly how SigNoz holds it together.

The primitives, and the foundation they sit on

The design rests on three moves that repeat in every track:

Record raw facts as OpenTelemetry, never a summary.
Grade those facts with a fail closed, three state verdict (PASS, BREACH, or UNKNOWN, never a comforting false zero).
Alert on the verdict in SigNoz, and where it is safe, act on it.

The foundation is deliberately boring and reproducible. SigNoz v0.133 runs on my
laptop, stood up with Foundry (WSL2 plus Docker), and the repo ships the exact
casting.yaml and casting.yaml.lock so a judge can re-run the same deployment.
The agent runs on a local Ollama model (qwen2.5:3b), no cloud and no API keys.
Every request emits an agent.invoke to llm.chat to tool.* span tree using the
OpenTelemetry GenAI semantic conventions, with tokens, cost, and latency dual
written to metrics.

In the loop, SigNoz plays four roles, and because all four read the same
telemetry, they share one source of truth: sensor, diagnostic surface (through the
SigNoz MCP server), scoreboard, and trigger.

Track 01: the self healing SRE sidekick

The first breach I chased is the retry tax. A fault drops the first completed LLM
response, so the agent runs the inference again and spends the tokens twice. It is
a perfect target: it is real telemetry (a duplicate llm.chat span tagged
retry.reason = "response_dropped"), it is measurable as an SLO (dropped divided
by total per cohort), and there is more than one valid fix, so the model has a
genuine decision to make.

One command runs the whole thing, and the entire cycle is a single distributed
trace.

Detection and verification are deterministic signoz_aggregate_traces queries. No
LLM is involved in judging. The model is only asked to decide what to do about a
confirmed breach. Its first move is a read_incident tool call, backed by two live
queries over the MCP server, and the structured evidence it gets back (a retry
rate, a dropped_and_retried count, and a root cause) comes straight out of the
traces the workload had just emitted.

Given that evidence, the local model chose disable_fault_injection on its own and
called it. Retry rate went from 40 percent to 0 percent, and every number in that
sentence is a SigNoz query, not a print statement.

A real alert is the trigger, not a cron job

The hands off path does not poll on a timer. A real SigNoz alert rule fires the
heal. When the retry rate crosses the SLO, SigNoz flips the rule to firing, a
bridge catches it over an Alertmanager webhook, opens a heal.trigger span, and
launches the governed loop. The same alert flipping back to resolved is how the
loop knows it is done. The alert and the heal are one distributed trace.

The part that makes it safe: a policy gate

Letting software act on production is only reasonable if the action is
constrained, and this is the bit most autonomous agent demos skip. Every actuator
mutation routes through a policy gate first. Autonomy runs on four levels (observe,
suggest, approve, auto), and each action carries a risk, a reversibility, and a
blast radius. In auto with a low risk cap, a low risk reversible action applies
itself, while a riskier action like switch_model is held for a human. Every
decision writes an audit line as a heal.policy metric, and if an applied fix does
not clear the breach, the loop restores a snapshot it took beforehand.

The second incident shows the gate earning its keep. A runaway agent stuck in a
loop runs up an unbounded bill. The sensor measures calls per request from the
traces against an SLO ceiling of 6, the model reads the runaway from SigNoz and
arms a set_cost_budget circuit breaker, and here is my favourite frame of the
whole project: the entire policy decision recorded on the span.

That breaker is a structural cut, not a warning. When a request crosses the budget
the agent stops issuing LLM calls and tags the span agent.request.severed = true.
Spend per request dropped from 0.000700 dollars to 0.000122 dollars, about an 82
percent cut, which the plug and play economics layer prices at roughly 1,734
dollars a month at 100,000 requests a day.

One more thing the loop learns: the fix is written to a verified memory in SigNoz,
so the next time the same incident recurs it heals from memory with no model call
at all, which is both faster and calibration free. Running it again while writing
this, a fresh 33 percent retry breach healed straight from memory (proven four
times over now), no model call, MTTR 149 seconds, trace
d0c14392999aef86af95032a8dd080e0.

Track 02: an MCP Contract Lab, three signals, and a zero code proxy

Track 02 is about instrumentation and dashboards, so I aimed the same primitives at
the Model Context Protocol itself.

mcp2_cert is an active probe that certifies an MCP server against a contract. It
exercises initialize, tools/list, and tools/call, grades each result with the
same three state verdict, and emits every exchange as an mcp.* span with a stable
fingerprint. Against my SigNoz MCP server it returns CERTIFIED with fingerprint
ca5859fcf24f5e03.

Then I built the passive companion: an always on instrumentation proxy
(mcp-proxy). It sits transparently between any MCP client and server and turns
their JSON RPC traffic into mcp.<method> spans and mcp.client.* metrics with
zero code changes to either side. To prove it is faithful, I drove the full
certification battery through the proxy. The result was still CERTIFIED with the
identical fingerprint ca5859fcf24f5e03, while the proxy independently emitted
correctly graded spans for every call. You can instrument an MCP integration you do
not own, and lose nothing.

All of that lands on a three signal dashboard (traces, metrics, and logs together)
built through the SigNoz v5 Query Builder API. Every panel self verifies against
the query_range API before the dashboard is created, so a panel that cannot
resolve never ships.

Track 03: energy, carbon, and accessibility as telemetry

Track 03 says observe anything, so I observed two things nobody usually treats as
telemetry.

WattTrace GreenOps measures joules and grams of CO2e per verified answer. It prices
every token to energy on a deterministic token basis, folds a run to its weakest
provenance (so a fallback estimate is never shown as a hardware reading), and
grades it fail closed against a budget. Running a control cohort against a retry
fault, the same verified answers cost 753 J against 1101 J, a 46 percent energy
regression, most of it pure waste on the retried calls. The retry tax, reframed as
a sustainability regression you can alert on.

AccessTrace applies the exact same fail closed primitives to web accessibility. A
real headless browser walks a WCAG journey (page load, navigation, main content,
signup form), runs axe-core scoped at each stage, and emits the journey as a trace:
access.suite to access.journey to access.step. The killer view is that the
access.step spans localise which part of the user flow breaks. On a broken demo
page the verdict is BREACH with a weighted debt of 40 (two critical and four
serious issues, at the exact failing stage), and its fixed twin scores 0. That is a
100 percent reduction, and two live SigNoz alerts fire on the breach.

The cross track finale: carbon becomes a heal sensor

This is where the three tracks stop being three projects. The WattTrace energy
verdict from Track 03 becomes a heal sensor for the Track 01 self healer, read over
the Track 02 MCP protocol. The self healer detects a carbon SLO breach, defined as
the share of a cohort's inference energy wasted on dropped and retried calls, and
closes it through the same governed loop.

The calibration behind that is the interesting part. A healthy multi call workload
sits around 867 J per answer, only about 4 percent under the 900 J single call
reference budget, so gating the verify step on the absolute cap would roll back a
perfectly good fix on ordinary token noise. Gating on wasted fraction instead is
calibration free: a healthy cohort wastes zero, so a verified fix is never rolled
back. In a live carbon heal the footprint per answer dropped from 0.3275 gCO2e to
0.1183 gCO2e (2649 J to 958 J), and the fix was learned to memory. One loop, three
tracks, one trace.

How I used SigNoz, concretely

Traces: the full agent.heal, watt.suite, access.suite, and mcp.* span trees, every step correlated under one trace id.
Metrics: heal.slo.breach, heal.mttr, heal.policy, watttrace.energy.consumed, watttrace.carbon.emitted, and more, dual written alongside the traces.
Logs: trace correlated, structured heal lifecycle logs, so the loop shows up in all three signals.
Query Builder v5: every detector and every dashboard panel is a programmatic SigNoz query_range query. Detection, diagnosis, and verification are all just SigNoz queries, so the loop is built on the query surface, not merely pointed at it.
Dashboards: five importable dashboards (self healing, three signal reliability, MCP Contract Lab, WattTrace GreenOps, and AccessTrace), each self verified before creation, with their uuids checked into the repo.
Alerts: ten live rules across the tracks, and a real alert is the trigger for the heal, not just a notification.
The SigNoz MCP server: the model reads its own incidents through it, and the proxy instruments it.
Service accounts and API keys for programmatic access, plus the Foundry casting.yaml and casting.yaml.lock so the whole thing reproduces.

The honest bits

The hackathon asks for real experience, so here is what actually bit me. Small
models need a firm hand to tool call: llama3.2:3b kept emitting a malformed tool
call, so I moved to qwen2.5:3b, and even then the prompt has to spell out that
read_incident only reads and must be followed by a remediation. Keep the LLM out
of the reliability hot path: detection and verification stay deterministic, the
model only decides, and SigNoz judges. The observer effect is real, so the healer
and the workload run as separate services and every query is scoped to a single
cohort tag. CPU inference is genuinely slow, 10 to 110 seconds per call, which
means the latency tail is real and MTTR actually means something. And everything is
fail closed: a run that cannot be judged is UNKNOWN, never a false PASS, because a
zero energy reading is a broken measurement, not free work.

Reproduce it

It is one command per incident, the repo ships the exact casting.yaml so you can
stand up the same SigNoz deployment, and 234 network free unit tests cover the pure
grading logic offline:
github.com/gajanangitte/observable-agent.

Observability that acts, across reliability, protocol, carbon, and accessibility.
It turns out the hard part was never the acting. It was proving, in telemetry, that
the fire is out, the bill is capped, the carbon is cut, and the front door is open
to everyone.

The agent that heals itself: closing the loop with SigNoz

Gajanan Gitte — Sun, 19 Jul 2026 06:17:17 +0000

This is the fourth and final part of a series where I gave a local Llama agent
real observability with SigNoz running on my own laptop. First I gave it
OpenTelemetry eyes,
then I measured the retry tax
its silent retries cost, then I wired in the
SigNoz MCP server so it could read its own traces.
This time it does more than watch itself. It heals itself.

Every "AI plus observability" demo I had seen stopped in the same place: a
beautiful trace of the thing going wrong. The dashboard lights up, the tokens are
gone, and then a human reads it and fixes it by hand.

Observability tells you the agent is on fire. It does not put the fire out.

So for the Agents of SigNoz hackathon I set myself a harder goal. Close the loop.
Detect a reliability breach, diagnose it, act on it, and then prove the fix worked.
All with a local model, all on a laptop, and all grounded in SigNoz rather than the
model's word. No cloud, no API keys.

Here is what I learned building it.

SigNoz as the sensor, not the dashboard

The whole design rests on one idea. SigNoz plays three roles in a single loop, and
because all three read the same telemetry, they share one source of truth.

Sensor. A detector asks SigNoz "did this rollout breach the SLO?" with a signoz_aggregate_traces query. Deterministic. No LLM involved.
Diagnostic surface. When there is a breach, the model reads the incident evidence out of the same traces, through the SigNoz MCP server.
Scoreboard. After the fix, the loop queries SigNoz again. Breach cleared means healed, and that is what sets the MTTR.

That last point is the one I care about most. Detection and verification are
deterministic SigNoz queries. The model is only asked to decide what to do about a
confirmed breach. If it hallucinates, the decision might be wrong, but the breach
and the healed verdict are always facts pulled from telemetry. That is the line
between a party trick and something you would actually let run.

Two services show up in SigNoz, exactly as they would in production: the
observable-agent workload (the thing that gets sick) and the self-healer (the
loop that watches it, decides, and acts).

Incident number one: the retry tax

The first breach I chased is the retry tax from part two of the series. A fault
drops the first completed LLM response, so the agent runs the inference again and
spends the tokens twice. It is a perfect target for self healing. It is real
telemetry (a duplicate llm.chat span tagged retry.reason = "response_dropped"),
it is measurable as an SLO (dropped divided by total per cohort), and there is more
than one way to fix it, so the model has a genuine decision to make.

One command, python self_heal.py, runs the whole thing, and the entire cycle is a
single distributed trace.

agent.heal                                 2.93 min   service = self-healer
  heal.canary.pre     32.9s   BREAK IT    roll out the workload under the fault
  heal.detect          2.2s   DETECT      3x signoz_aggregate_traces (via MCP)
  heal.decide         51.7s   DIAGNOSE and DECIDE  (local qwen2.5:3b)
     llm.chat            26.5s
     tool.read_incident   1.6s   2x signoz_aggregate_traces  (evidence via MCP)
     llm.chat            13.4s
     tool.disable_fault_injection   2ms   the remediation, chosen by the model
  heal.canary.post    86.0s   VERIFY      roll out again under the fixed config
  heal.verify          2.3s   3x signoz_aggregate_traces

The model's first move is a read_incident tool call, backed by two live
signoz_aggregate_traces calls over MCP. The structured evidence it gets back,
retry rate 0.333, dropped_and_retried: 2, and a root cause, comes straight out
of the traces the workload had just emitted.

Given that, qwen2.5:3b chose disable_fault_injection on its own and called it.
Retry rate went from 40% to 0%, MTTR 141 seconds, and every number in that sentence
is a SigNoz query, not a print statement.

The part that makes it safe: a policy gate

Letting software act on production is only reasonable if the action is constrained.
This is the bit most "autonomous agent" demos skip, and it is the bit I think
actually matters. So every actuator mutation routes through a policy gate before it
can touch anything.

Autonomy levels: observe, suggest, approve, auto. In auto the gate applies
changes itself. In approve it holds them for a human. In suggest or observe it
only proposes.

Per action risk, reversibility, and blast radius. In auto with a low risk cap, low
risk reversible actions apply automatically. A riskier action like switch_model is
held for approval instead of being forced through.

An audit line per decision: allowed, held, or denied, with the reason, recorded as a
heal.policy metric.

And when an applied fix does not clear the breach, the loop rolls back. It snapshots
the control plane before each attempt and restores it on a failed verify. The retry
incident heals on the first try, so it never needs the rollback, but the machinery
is there, and it is what separates acting from acting safely.

Incident number two: the bill shock kill switch

The retry tax wastes tokens a few at a time. The failure that actually scares
finance teams is a runaway agent, one stuck in a loop, issuing LLM call after LLM
call, running up an unbounded bill. Same governed loop, a different sensor and a
different fix.

python self_heal.py --scenario cost

The sensor measures calls per request from the traces, with an SLO ceiling of 6. The
model read the runaway from SigNoz, "28 llm.chat calls across 2 requests, 14 per
request, versus a max of 6", and chose set_cost_budget, arming a per request cost
circuit breaker.

That breaker is a real structural cut, not a warning. When a request's cumulative
cost crosses the budget, the agent stops issuing LLM calls and tags the span
agent.request.severed = true. Spend per request dropped from $0.000700 to
$0.000123, about an 82% cut, calls per request from 13.5 to 2.5, MTTR 177 seconds.

And here is my favourite frame of the whole project, the set_cost_budget span,
with the entire policy decision recorded right on it.

heal.policy.risk = low, heal.policy.reversible = true, heal.policy.autonomy = auto, so heal.policy.allow = true. The fix is low risk and reversible, so the gate
lets it through automatically. A riskier action would have stopped and waited for a
human. That is the difference between an agent that can act and one you would let.

The honest bits

The hackathon asks for real experience, so here is what actually bit me.

Small models need a firm hand to tool call. llama3.2:3b kept emitting a malformed
tool call. qwen2.5:3b, the same size, does reliable OpenAI format tool calling on
Ollama, so I switched. Even then, the prompt has to spell out that read_incident
only reads and must be followed by a remediation, and the output token cap has to
leave room for the decision and the tool call, or the remediation gets truncated.

Keep the LLM out of the reliability hot path. Detection and verification are
deterministic. The model decides. SigNoz judges.

The observer effect is real. The healer's own spans would pollute the workload's SLO
math, so the two run as separate services and every query is scoped to a single
cohort tag.

CPU inference is slow, and that is honest. Each llm.chat runs 10 to 110 seconds on
CPU, so a full heal takes a few minutes. The latency tail is real, which means there
is genuinely something to observe, and MTTR means something.

Why I built it on SigNoz

I could not have built this on a dashboard I could only look at. I needed a backend I
could query programmatically, inside the loop, and one the model could reach through
a standard interface. SigNoz gave me both, the trace store and the MCP server,
runnable on my own laptop, for free. Detection, diagnosis, and verification are all
just SigNoz queries. The loop is built on SigNoz's query surface, not merely pointed
at it.

If you want to run it, it is one command per incident, and the repo ships the exact
casting.yaml so you can stand up the same SigNoz deployment:
github.com/gajanangitte/observable-agent.

Observability that acts. Turns out the hard part is not the acting. It is proving, in
telemetry, that the fire is actually out.

I gave my local Llama agent the SigNoz MCP server and asked it to debug itself

Gajanan Gitte — Sat, 18 Jul 2026 15:00:37 +0000

The agent that reads its own traces

Observability is something we do to our agents. We instrument them, then a human opens SigNoz and reads the waterfall. But SigNoz now ships an MCP server, a Model Context Protocol interface that turns "search traces", "aggregate latency", "list services" into tools an LLM can call. So the console isn't human only anymore. Which raises a fun question:

What if the agent read its own traces and debugged itself?

So I connected my agent, a tiny qwen2.5:3b running on CPU via Ollama, no GPU, no cloud, to the SigNoz MCP server and asked it one question: "Why are you slow?" It called an MCP tool, pulled its own p95 latency out of SigNoz, and answered:

"The llm.chat model calls have a p95 latency of approximately **101,375.3 ms, while the tool. calls have a p95 of about 1.5 ms. The model call is roughly 67,584 times slower than the tool call... focus on optimizing the llm.chat model calls or consider a more efficient alternative."*

That is the agent rediscovering, from the inside, the exact lesson I found by hand in my first post (one LLM call was 84.5% of a request). Only this time I didn't read the trace. It did.

Everything is local and free: SigNoz v0.133, the SigNoz MCP server, and Ollama on CPU. Here's how it works, and the two things that actually made a tiny model reliable over MCP.

The loop: an agent that observes the observer

The agent already emitted OpenTelemetry traces (agent.invoke → llm.chat → tool.*). To let it read them back, I gave it a second entrypoint with a different toolset: instead of the fake SRE tools, it gets SigNoz MCP tools, and the whole session is rooted in an agent.introspect span. That means the self debugging run is itself a trace you can open in SigNoz, the observer, observed:

Top to bottom: agent.introspect → llm.chat (the model decides what to look at) → tool.latency_by_operation → mcp.signoz_aggregate_traces (a real MCP call out to the server) → llm.chat (the diagnosis). The agent made an MCP request to SigNoz as a step inside its own trace.

The MCP server itself is one binary in HTTP mode:

SIGNOZ_URL=http://localhost:8080 SIGNOZ_API_KEY=$KEY \
  TRANSPORT_MODE=http MCP_SERVER_PORT=8000 ./signoz-mcp-server
# -> listening on :8000/mcp, 41 tools registered

And the client bridge is deliberately tiny. The agent loop is synchronous, so each tool call opens a short lived streamable HTTP session:

from mcp import ClientSession
from mcp.client.streamable_http import streamablehttp_client

class SigNozMCP:
    def call_tool(self, name, args):
        async def run():
            async with streamablehttp_client(self.url) as (r, w, _):
                async with ClientSession(r, w) as s:
                    await s.initialize()
                    return await s.call_tool(name, args or {})
        result = asyncio.run(run())
        return "\n".join(c.text for c in result.content if c.type == "text")

Every tool the model calls is wrapped in its own mcp.* span, so the MCP round trip shows up in the trace with the exact request it made:

def _mcp_call(mcp, tool_name, args):
    with tracer.start_as_current_span(f"mcp.{tool_name}", kind=SpanKind.CLIENT) as span:
        span.set_attribute("mcp.tool.name", tool_name)
        span.set_attribute("mcp.method", "tools/call")
        span.set_attribute("mcp.transport", "streamable_http")
        span.set_attribute("mcp.request.args", json.dumps(args))
        return json.loads(mcp.call_tool(tool_name, args))

Click that span and there's the proof it was a genuine MCP protocol call, and that the agent was querying itself:

mcp.tool.name: signoz_aggregate_traces, mcp.method: tools/call, and the args: aggregate p95 of duration_nano, grouped by span name, filtered to service.name = 'observable-agent'. The agent asked SigNoz for its own latency profile.

Honest problem #1: a 3B model can't drive 41 raw MCP tools

Here's the part the demos skip. The SigNoz MCP server exposes 41 tools, and the good ones take rich arguments: nanosecond durations, free form SigNoz filter expressions, groupBy keys, relative time ranges. That's fine for Claude. It is not fine for a 3B model on CPU. My first attempt handed the model the raw tools. llama3.2:3b didn't even emit a valid tool call. It printed one as text:

A: {"name":"latency_by_operation","parameters:{}}   # <- malformed, not a real tool call

Two fixes made it reliable:

A tool calling model. llama3.2:3b fumbles the function calling format; qwen2.5:3b (same size, same CPU) gets it right. Model choice matters more than model size here.
Curated, argument free tools. Instead of exposing 41 tools with complex schemas, I expose three with no arguments: list_my_services, latency_by_operation, find_slowest_traces, each of which pins the correct arguments and calls a real MCP tool under the hood. The model picks what to look at; the wrapper handles how to ask.

def latency_by_operation(**_):
    """MCP: signoz_aggregate_traces -> p95 latency per operation."""
    parsed = _mcp_call(mcp, "signoz_aggregate_traces", {
        "aggregation": "p95", "aggregateOn": "duration_nano",
        "groupBy": "name", "filter": f"service.name = '{SERVICE}'",
        "timeRange": "6h",
    })
    # ...trim the big JSON down to [{operation, p95_ms}] the model can reason about

This is the pattern for small models + MCP: don't expose the raw firehose; expose a few intention shaped tools and let the server do the heavy lifting.

Honest problem #2: small models hallucinate numbers, so let SigNoz do the math

Even with a clean tool result, qwen2.5:3b's first diagnosis invented figures out of thin air: "llm.chat p95 is 200ms, tools are 300ms." Those numbers appear nowhere in the data. A 3B model is a bad calculator and a worse transcriber of long decimals.

The fix isn't a bigger model. It's the right division of labour. Percentile math is exactly what SigNoz's aggregation is for, so I compute the comparison in the tool and hand the model a factual, hard to hallucinate headline. The model's job is to orchestrate and interpret, not to do nanosecond arithmetic in its head:

headline = (f"llm.chat p95 = {llm_ms} ms; slowest tool ({tool_name}) "
            f"p95 = {tool_ms} ms; the model call is ~{ratio}x slower than the tool call.")
return {"headline": headline, "operations": ops}

With that grounding, the diagnosis became correct and specific. You can see the exact numbers it read, captured in the tool span's result attribute:

llm.chat p95 101,375.3 ms vs the slowest real tool at 1.5 ms. The agent read that and concluded, correctly, that it is ~67,000× bottlenecked on the model, not its logic, not its tools.

The observer effect is real (and it's in the trace)

One honest wrinkle: the introspection runs are themselves slow (CPU inference), so they pollute the very data the agent reads. The first version of my latency tool proudly reported that agent.introspect was the slowest operation because it was measuring itself. The mcp.* and telemetry reading tool.* spans, dominated by the SigNoz round trip, leaked into the "tool" numbers too and shrank the ratio from ~67,000× to a meaningless 12×.

The fix was to exclude the introspection scaffolding (agent.introspect, mcp.*, the three reader tools) from the comparison so the agent reasons about its real request serving workload. But it's a genuine lesson worth stating plainly: when an agent observes itself, it perturbs what it observes. You can see it happen span by span.

Why this is a different corner of MCP

SigNoz's own MCP examples, reasonably, show a frontier cloud model (Claude, Cursor) querying a production stack. This is the opposite extreme on purpose:

	Typical MCP demo	This
Model	Cloud frontier (Claude/GPT)	`qwen2.5:3b`, CPU only, local
Target	Someone else's production	Its own traces
Network	SaaS	Fully offline
Point	Human asks about a service	Agent debugs itself

None of it is cloud shaped. A 3 billion parameter model on a laptop CPU used a standards based protocol to read its own OpenTelemetry data out of a self hosted backend and reached a true conclusion about its own performance. That is a self observing agent, built from open parts.

What's honest about this

The model doesn't do the percentile math and I curated its tools. That's not cheating, it's system design: the LLM orchestrates and interprets, SigNoz computes, MCP is the contract between them. What's completely real is the loop: the agent chose to look at latency, issued a real tools/call to the SigNoz MCP server, read numbers measured from its own spans, and diagnosed itself. And it did it on hardware you already own.

The takeaway from my first and second posts was "you can't own an agent you can't observe." MCP adds a sequel: once the telemetry is an agent callable API, the agent can observe itself. The next step, closing the loop from diagnose to act, is suddenly a very short walk.

Stack: SigNoz v0.133 (self hosted, WSL2 + Docker Engine), SigNoz MCP server (HTTP/streamable transport, 41 tools), OpenTelemetry Python SDK, Ollama qwen2.5:3b on CPU. Introspection agent + MCP bridge + curated tools: ~200 lines of Python. Built for the WeMakeDevs × SigNoz "Agents of SigNoz" hackathon.

The Retry Tax: what a local Llama agent's silent retries actually cost, measured in self hosted SigNoz

Gajanan Gitte — Sat, 18 Jul 2026 15:00:33 +0000

The Retry Tax

Every "AI agent" retries. The model returns malformed JSON, a guardrail rejects the answer, a tool call schema doesn't validate, a request times out, so the agent quietly runs the model again. That retry is usually invisible. It doesn't throw. The user still gets an answer. But you paid for the inference twice: twice the tokens, twice the latency, twice the GPU seconds. I call it the retry tax, and until you trace it, it's a silent line item on your bill.

In an earlier post I gave a small local Llama SRE agent OpenTelemetry eyes and found one LLM call was 84.5% of a request. This post is the follow up I actually care about: I made that agent retry once, deliberately, and measured the tax in self hosted SigNoz. One wasted call, a response the agent generated and then threw away, turned out to be 34% of the request's wall clock time. I only know that because the trace shows the work that got discarded.

Everything here is local and free: SigNoz v0.133, Ollama running llama3.2:1b on CPU, and ~200 lines of instrumented Python. No cloud model, no vendor dashboard.

The failure I wanted to see

Real retries have messy causes. To study the cost of a retry without the noise of why, I injected a clean, deterministic fault: drop the first completed response exactly once per request. The key word is completed: the model already ran, the tokens were already spent. Then the agent has to infer again. That's the honest shape of a retry: you don't get the tokens back.

The whole fault is four environment driven knobs in config.py:

# EXPERIMENT_ID tags every span + metric so "control" and "chaos" cohorts are
# directly comparable in SigNoz. CHAOS_DROP_RESPONSE_ONCE drops the first
# completed response of each request, forcing exactly one retry.
EXPERIMENT_ID   = os.getenv("EXPERIMENT_ID", "")
CHAOS_DROP_ONCE = os.getenv("CHAOS_DROP_RESPONSE_ONCE", "0") == "1"
LLM_MAX_ATTEMPTS = int(os.getenv("LLM_MAX_ATTEMPTS", "1"))
RETRY_BACKOFF_MS = int(os.getenv("RETRY_BACKOFF_MS", "250"))

The agent's LLM round trip becomes an attempt loop. Each attempt is its own llm.chat span, so a retry literally duplicates the span inside the same trace: the retry tax made visible:

def _chat(self, messages):
    max_attempts = max(config.LLM_MAX_ATTEMPTS, 2 if config.CHAOS_DROP_ONCE else 1)
    for attempt in range(1, max_attempts + 1):
        with tracer.start_as_current_span("llm.chat", kind=SpanKind.CLIENT) as span:
            span.set_attribute("llm.attempt", attempt)
            resp = self.client.chat.completions.create(...)   # inference runs here
            in_tok  = resp.usage.prompt_tokens
            out_tok = resp.usage.completion_tokens

            # Injected fault: drop this completed response exactly once. The
            # tokens above were really generated, so we still record them
            # (status="dropped") -- that is the wasted work of the retry.
            if attempt < max_attempts and getattr(self._chaos, "armed", False):
                self._chaos.armed = False
                telemetry.record_llm(config.MODEL, in_tok, out_tok, latency_ms, "dropped")
                telemetry.record_retry(config.MODEL, "response_dropped")
                span.set_attribute("fault.injected", True)
                span.set_attribute("retry.reason", "response_dropped")
                span.add_event("retry.scheduled",
                               {"retry.reason": "response_dropped",
                                "retry.backoff_ms": backoff_ms,
                                "next.attempt": attempt + 1})
                span.set_status(Status(StatusCode.ERROR, "response dropped after completion"))
                time.sleep(backoff_ms / 1000.0)
                continue
            ...
            return choice.message

Two details make this trace useful instead of just red:

The dropped attempt still records its tokens (status="dropped"). Wasted work you don't measure isn't waste, it's a mystery.
experiment.id is stamped on every span and every metric, so a control run and a chaos run are directly comparable in the Query Builder. One counter counts the retries themselves:

def record_retry(model, reason):
    """Count one retry (e.g. a dropped response forced the agent to re-infer)."""
    _retries.add(1, _tag({"gen_ai.request.model": model, "retry.reason": reason}))

Two cohorts, same ten questions

I ran the identical load twice, ten SRE questions like "Is checkout healthy?", and tagged them:

# control: no chaos
EXPERIMENT_ID=rt-control python run_load.py

# chaos: drop the first response of every request, once
EXPERIMENT_ID=rt-chaos CHAOS_DROP_RESPONSE_ONCE=1 python run_load.py

Then I asked ClickHouse (SigNoz's store) what the tax was:

Metric (10 requests each)	`rt-control`	`rt-chaos`	Δ
`llm.chat` spans	20	30	+50%
Dropped (wasted) calls	0	10	+10
Total tokens	7,322	11,515	+57%
of which wasted	0	4,393	38% of all tokens
Cost (USD)	$0.000734	$0.001153	+57%
Avg latency / request	43.1 s	57.0 s	+32%
p95 latency	91.2 s	106.1 s	+16%

One retry per request, the mildest possible retry policy, cost 57% more tokens and money and made the agent a third slower. 38% of every token the chaos cohort burned went to answers it threw in the bin. Now imagine a real policy of "retry up to 3× on invalid JSON" against a paid API.

Reading the tax in a single trace

Here's one chaos request. The waterfall tells the whole story at a glance: 5 spans, 1 error:

Top to bottom: agent.invoke → a red llm.chat (the dropped attempt) → a second llm.chat (the retry) → tool.get_service_health → a final llm.chat. Two back to back inference spans before any useful work happens. In the clean cohort that first red span simply doesn't exist:

Click the red span and the attributes make the waste concrete, and queryable:

That single dropped call was 11.12s, 34.08% of total execution time, spent 431 tokens (413 in + 18 out), and carries a retry.scheduled span event with retry.backoff_ms: 250 and next.attempt: 2. Because these are real span attributes, "show me every dropped inference and how many tokens it wasted" is just a filter: name = 'llm.chat' AND retry.reason = 'response_dropped'.

A dashboard for the tax

Same filter, turned into a Retry Tax dashboard via the SigNoz dashboards API (v5 query builder schema, dataSource: traces). The experiment.id tag does the control vs chaos split for free:

The clearest panel is Cost by Cohort: the pink rt-chaos slice ($0.00115) is over 1.5× the green rt-control slice ($0.00073) for the exact same ten questions. The value tiles count retried calls and wasted tokens straight from the span attributes, no code change, just a query. (They read a little above the clean table further up because the dashboard tallies every request in its window, warm ups included; the table is the controlled 10 vs 10.)

Closing the loop: alert on the tax

A number on a dashboard nobody looks at is worthless. So I made the retry tax page me. SigNoz v0.133 alert rules use the new v5 queries: [] array (each query wrapped in a {type, spec} envelope). This is worth knowing, because the old builderQueries map returns a flat "alert rule is not valid". The rule counts dropped attempts over a window:

{
  "alert": "Agent Retry Tax: LLM responses being retried",
  "alertType": "TRACES_BASED_ALERT",
  "version": "v5",
  "condition": {
    "compositeQuery": {
      "queries": [{
        "type": "builder_query",
        "spec": {
          "name": "A", "signal": "traces",
          "aggregations": [{ "expression": "count()" }],
          "filter": { "expression":
            "service.name = 'observable-agent' AND name = 'llm.chat' AND retry.reason = 'response_dropped'" }
        }
      }],
      "panelType": "graph", "queryType": "builder"
    },
    "op": "above", "target": 0, "matchType": "at_least_once"
  }
}

POST /api/v1/rules, and within a minute it was firing on the chaos data:

Now the moment my agent starts silently inferring again, dropped responses, guardrail rejects, 429 backoffs, I find out from an alert instead of from a surprise invoice.

What's honest about this

I injected the fault, so don't read the exact 57% as a universal constant. Your tax depends on your retry policy and how often reality trips it. But the mechanism is completely real: an LLM call that completes and then gets discarded costs you full tokens and full latency, and by default that cost is invisible because the agent recovers. Genuine sources of the same tax: JSON that won't parse, tool arguments that fail schema validation, content filter rejections, timeouts, and rate limit retries. Each invokes the model again, producing the same duplicated llm.chat span you see here. (This demo wires up the dropped response path; the others share the identical signature.)

The takeaway isn't "retries are bad". Retries are how agents stay reliable. It's that a retry you do not observe is a tax you can't see, can't bound, and can't alert on. Three OpenTelemetry attributes, llm.attempt, retry.reason, and the token counts for the dropped attempt, turn it into a first class signal you can trace, chart, and page on. In SigNoz, that took one attribute loop and one filter.

If you can't observe your AI agents, you don't own them, and you definitely don't own their retry bill.

Stack: SigNoz v0.133 (self hosted via Foundry on WSL2 + Docker Engine), OpenTelemetry Python SDK, Ollama llama3.2:1b. Agent + instrumentation + load driver: ~250 lines of Python. Built for the WeMakeDevs × SigNoz "Agents of SigNoz" hackathon.

I gave a local Llama agent OpenTelemetry eyes: tracing tokens, cost, and the latency tail in self hosted SigNoz

Gajanan Gitte — Sat, 18 Jul 2026 15:00:28 +0000

I gave a local Llama agent OpenTelemetry eyes

"If you can't observe your AI agents, you don't own them."

That line from the Agents of SigNoz hackathon stuck with me, so I tested it literally. I took a small SRE "sidekick" agent running on a local Llama model, with no OpenAI bill and no vendor dashboard, wired it up with OpenTelemetry, and pointed it at a self hosted SigNoz. Then I watched what an agent actually does when you ask it a question.

The short version: the tool calls were never the problem. One LLM call was 84.5% of the entire request. I only know that number because the agent had eyes.

This post is the narrow, practical walkthrough: self host SigNoz, instrument an agent with the OpenTelemetry GenAI semantic conventions, and read the trace, tokens, cost, and latency tail that come out the other side.

The stack (all local, all free)

Piece	What I used
Observability backend	SigNoz v0.133 (self hosted via Foundry)
Telemetry	OpenTelemetry Python SDK (manual spans + metrics + logs)
Transport	OTLP/HTTP → `:4318`
LLM	Ollama running `llama3.2:1b` and `llama3.2:3b` on CPU
Agent	~200 lines of Python: a tool calling loop

Heads up if you're following an old tutorial: the install.sh script and the bundled docker-compose files are deprecated as of v0.130. SigNoz now installs through Foundry, a small "observability stack as code" CLI. It's still three steps:

# 1. install the Foundry CLI
curl -fsSL https://signoz.io/foundry.sh | bash

# 2. a five line casting.yaml (this is the whole file)
#   apiVersion: v1alpha1
#   kind: Installation
#   metadata: { name: signoz }
#   spec: { deployment: { flavor: compose, mode: docker } }

# 3. deploy
foundryctl cast -f casting.yaml

I'm on Windows, so I ran everything inside WSL 2 with Docker Engine, not Docker Desktop. That's not a personal quirk: SigNoz's own docs warn that ClickHouse Keeper crash loops (exit 139, segfaults) under Docker Desktop's VM layer on Windows. I hit exactly that before switching, so consider this your warning too.

A minute later the UI is on http://localhost:8080, and its onboarding checklist lights up green the moment your app starts sending data:

Three ticks, logs, traces, and metrics ingestion active, and my service observable-agent shows up in the services list. That's the whole "did my telemetry land?" question answered in one screen.

The agent: a tiny SRE sidekick

The agent answers on call questions like "Is checkout healthy?" or "What's my error budget?". It has four tools:

get_service_health(service)
list_recent_deploys(service)
calculate_error_budget(slo_percent, actual_error_rate)
search_runbook(topic)

It runs the classic agent loop: send the question + tool schemas to the model, let the model pick tools, run them, feed results back, and let the model write the final answer. Nothing exotic. That is exactly the point. This is the shape of most "AI agents" in production.

What makes it observable is a three level span tree:

agent.invoke        (SERVER)   ← one per user question
└─ llm.chat         (CLIENT)   ← each model round trip
└─ tool.<name>      (INTERNAL) ← each tool execution

Instrumenting the LLM call the "right" way

OpenTelemetry has GenAI semantic conventions: a standard set of gen_ai.* attributes so any backend can understand an LLM call. Instead of inventing my own keys, I set the standard ones on each llm.chat span:

with tracer.start_as_current_span("llm.chat", kind=SpanKind.CLIENT) as span:
    span.set_attribute("gen_ai.system", "ollama")
    span.set_attribute("gen_ai.operation.name", "chat")
    span.set_attribute("gen_ai.request.model", model)

    resp = client.chat.completions.create(model=model, messages=msgs, tools=TOOL_SCHEMAS)

    span.set_attribute("gen_ai.response.finish_reason", resp.choices[0].finish_reason)
    # one helper writes tokens + cost to BOTH the span and the metrics:
    record_llm(model, resp.usage.prompt_tokens, resp.usage.completion_tokens, latency_ms)

That record_llm helper is where the trick lives: the same numbers land on the span and on OpenTelemetry metrics in one place:

def record_llm(model, in_tok, out_tok, latency_ms):
    cost = cost_usd(model, in_tok, out_tok)
    attrs = {"gen_ai.request.model": model, "gen_ai.system": "ollama"}
    token_counter.add(in_tok,  {**attrs, "gen_ai.token.type": "input"})
    token_counter.add(out_tok, {**attrs, "gen_ai.token.type": "output"})
    cost_counter.add(cost, attrs)
    llm_latency.record(latency_ms, attrs)
    span = trace.get_current_span()                 # the live llm.chat span
    span.set_attribute("gen_ai.usage.input_tokens",  in_tok)
    span.set_attribute("gen_ai.usage.output_tokens", out_tok)
    span.set_attribute("gen_ai.usage.cost_usd",      round(cost, 6))

So one call feeds two questions: "what happened in this exact request?" (the span) and "what's happening across all of them?" (the metrics gen_ai.client.token.usage, gen_ai.client.cost, gen_ai.client.operation.duration).

That's it. No agent framework, no auto instrumentation magic, just the standard conventions, so SigNoz knows exactly what it's looking at.

A note on the spec (I checked the docs, as the guide advises). The OpenTelemetry GenAI semantic conventions are at Development stability and recently moved to their own repo. A few honest deviations in the code above: gen_ai.system is now deprecated in favour of gen_ai.provider.name; gen_ai.usage.input_tokens/output_tokens are the current names (the prompt_tokens/completion_tokens you see are the OpenAI SDK's field names, which I map onto the span); and gen_ai.response.finish_reasons is officially plural (a string array). gen_ai.usage.total_tokens, gen_ai.usage.cost_usd, and the gen_ai.client.cost metric are my own extensions. The spec has no cost signal yet. The spec also models gen_ai.client.token.usage as a Histogram and gen_ai.client.operation.duration in seconds; I kept a counter and milliseconds for readability. And per the span guide the recommended span name is {operation} {model} (e.g. chat llama3.2:1b). I use llm.chat because it reads cleaner in a waterfall.

Reading the trace: where did the time actually go?

Here's a single question, "Inventory feels slow. Pull its health, recent deploys, and the right runbook.", as one distributed trace in SigNoz:

Six spans, zero errors. Read top to bottom:

agent.invoke: the whole request, 1.83 min.
llm.chat (17 s): the model reads the question and decides which tools to call.
tool.get_service_health, tool.list_recent_deploys, tool.search_runbook: three tools, each ~1 ms.
llm.chat (1.54 min): the model synthesizes the final answer from the tool output.

The tools are a rounding error. The first LLM call is quick. The final synthesis call is the whole story, and SigNoz labels it for you when you click the span:

1.54 mins, 84.51% of total exec time. Right there in the drawer are the GenAI attributes I set: gen_ai.request.model = llama3.2:1b, input_tokens = 286, output_tokens = 120, total_tokens = 406, gen_ai.usage.cost_usd, and finish_reason = stop. SigNoz even captured the completion text as a span event.

This is the payoff of the semantic conventions: I didn't build a custom "LLM view." I set standard attributes, and the generic trace UI became an LLM debugger. The lesson, the long context generation, not the tools, owns your p95, is the kind of thing you'd never guess from logs alone.

From one trace to a fleet: the dashboard

One trace is a debugging tool. To own the agent I need aggregates, so I built an LLM & Agent Observability dashboard straight from the same span attributes:

Across a short load run of varied SRE questions:

20 LLM calls · 6,064 input tokens · 1,439 output tokens · $0.00075 total cost
Token usage over time, split into input vs. output: input dominates because every tool result is re fed into the prompt.
LLM latency p95/p99 by model: the tail climbs past 1.6 min on CPU, which is precisely the cold, uncomfortable truth you want on a dashboard instead of in a user complaint.
Span latency p95 by operation: the single clearest panel on the board: agent.invoke and llm.chat ride high while every tool.* span sits pinned to the floor. The waterfall lesson, now true across the whole fleet.

About that cost: locally it's essentially zero (I compute it from a per model price table). But the instrumentation is identical to what you'd use against a paid API. Point the provider at openai or anthropic instead of ollama and the same dashboard is now watching real dollars, before the invoice does.

My favorite SigNoz feature

The trace → span drawer won me over. Because SigNoz speaks the OpenTelemetry GenAI conventions natively, a plain span turned into a per call LLM inspector: model, tokens, cost, finish reason, and the completion event, with zero custom UI. Add the "% of total exec time" on each span and root causing a slow agent stops being guesswork.

Runner up: the metrics query builder let me group the same gen_ai.* data by model and token type without writing SQL.

Honest takeaways

Instrument the standard, not the vendor. OpenTelemetry GenAI attributes made a generic backend understand my agent. No lock in, no framework.
Agents hide their latency in one span. My tools were ~1 ms; a single synthesis call was 84.5%. You cannot fix what you cannot see.
Self hosting SigNoz is a five line casting.yaml and one foundryctl cast. Traces, metrics, and logs in one place, on my laptop, for free.
Local models make great observability test rigs. CPU inference is slow, which means the interesting tail exists, so you actually have something to observe.

Next, I'm turning this into a Track 01 project: a self healing SRE sidekick that reads its own SigNoz traces to decide what to do. But that's another post.

Everything here, the agent, the OpenTelemetry wiring, and the dashboard JSON, is reproducible; the commands above are the ones I actually ran.