DEV Community: Gajanan Gitte

I gave my local Llama agent the SigNoz MCP server and asked it to debug itself

Gajanan Gitte — Sat, 18 Jul 2026 15:00:37 +0000

The agent that reads its own traces

Observability is something we do to our agents. We instrument them, then a human opens SigNoz and reads the waterfall. But SigNoz now ships an MCP server — a Model Context Protocol interface that turns "search traces", "aggregate latency", "list services" into tools an LLM can call. So the console isn't human-only anymore. Which raises a fun question:

What if the agent read its own traces and debugged itself?

So I connected my agent — a tiny qwen2.5:3b running on CPU via Ollama, no GPU, no cloud — to the SigNoz MCP server and asked it one question: "Why are you slow?" It called an MCP tool, pulled its own p95 latency out of SigNoz, and answered:

"The llm.chat model calls have a p95 latency of approximately **101,375.3 ms, while the tool. calls have a p95 of about 1.5 ms. The model call is roughly 67,584 times slower than the tool call... focus on optimizing the llm.chat model calls or consider a more efficient alternative."*

That is the agent re-discovering, from the inside, the exact lesson I found by hand in my first post (one LLM call was 84.5% of a request). Only this time I didn't read the trace. It did.

Everything is local and free: SigNoz v0.133, the SigNoz MCP server, and Ollama on CPU. Here's how it works — and the two things that actually made a tiny model reliable over MCP.

The loop: an agent that observes the observer

The agent already emitted OpenTelemetry traces (agent.invoke → llm.chat → tool.*). To let it read them back, I gave it a second entrypoint with a different toolset: instead of the fake SRE tools, it gets SigNoz MCP tools, and the whole session is rooted in an agent.introspect span. That means the self-debugging run is itself a trace you can open in SigNoz — the observer, observed:

Top to bottom: agent.introspect → llm.chat (the model decides what to look at) → tool.latency_by_operation → mcp.signoz_aggregate_traces (a real MCP call out to the server) → llm.chat (the diagnosis). The agent made an MCP request to SigNoz as a step inside its own trace.

The MCP server itself is one binary in HTTP mode:

SIGNOZ_URL=http://localhost:8080 SIGNOZ_API_KEY=$KEY \
  TRANSPORT_MODE=http MCP_SERVER_PORT=8000 ./signoz-mcp-server
# -> listening on :8000/mcp, 41 tools registered

And the client bridge is deliberately tiny — the agent loop is synchronous, so each tool call opens a short-lived streamable-HTTP session:

from mcp import ClientSession
from mcp.client.streamable_http import streamablehttp_client

class SigNozMCP:
    def call_tool(self, name, args):
        async def run():
            async with streamablehttp_client(self.url) as (r, w, _):
                async with ClientSession(r, w) as s:
                    await s.initialize()
                    return await s.call_tool(name, args or {})
        result = asyncio.run(run())
        return "\n".join(c.text for c in result.content if c.type == "text")

Every tool the model calls is wrapped in its own mcp.* span, so the MCP round-trip shows up in the trace with the exact request it made:

def _mcp_call(mcp, tool_name, args):
    with tracer.start_as_current_span(f"mcp.{tool_name}", kind=SpanKind.CLIENT) as span:
        span.set_attribute("mcp.tool.name", tool_name)
        span.set_attribute("mcp.method", "tools/call")
        span.set_attribute("mcp.transport", "streamable_http")
        span.set_attribute("mcp.request.args", json.dumps(args))
        return json.loads(mcp.call_tool(tool_name, args))

Click that span and there's the proof it was a genuine MCP protocol call — and that the agent was querying itself:

mcp.tool.name: signoz_aggregate_traces, mcp.method: tools/call, and the args: aggregate p95 of duration_nano, grouped by span name, filtered to service.name = 'observable-agent'. The agent asked SigNoz for its own latency profile.

Honest problem #1: a 3B model can't drive 41 raw MCP tools

Here's the part the demos skip. The SigNoz MCP server exposes 41 tools, and the good ones take rich arguments — nanosecond durations, free-form SigNoz filter expressions, groupBy keys, relative time ranges. That's fine for Claude. It is not fine for a 3B model on CPU. My first attempt handed the model the raw tools. llama3.2:3b didn't even emit a valid tool call — it printed one as text:

A: {"name":"latency_by_operation","parameters:{}}   # <- malformed, not a real tool call

Two fixes made it reliable:

A tool-calling model. llama3.2:3b fumbles the function-calling format; qwen2.5:3b (same size, same CPU) gets it right. Model choice matters more than model size here.
Curated, argument-free tools. Instead of exposing 41 tools with complex schemas, I expose three with no arguments — list_my_services, latency_by_operation, find_slowest_traces — each of which pins the correct arguments and calls a real MCP tool under the hood. The model picks what to look at; the wrapper handles how to ask.

def latency_by_operation(**_):
    """MCP: signoz_aggregate_traces -> p95 latency per operation."""
    parsed = _mcp_call(mcp, "signoz_aggregate_traces", {
        "aggregation": "p95", "aggregateOn": "duration_nano",
        "groupBy": "name", "filter": f"service.name = '{SERVICE}'",
        "timeRange": "6h",
    })
    # ...trim the big JSON down to [{operation, p95_ms}] the model can reason about

This is the pattern for small models + MCP: don't expose the raw firehose; expose a few intention-shaped tools and let the server do the heavy lifting.

Honest problem #2: small models hallucinate numbers, so let SigNoz do the math

Even with a clean tool result, qwen2.5:3b's first diagnosis invented figures out of thin air — "llm.chat p95 is 200ms, tools are 300ms" — numbers that appear nowhere in the data. A 3B model is a bad calculator and a worse transcriber of long decimals.

The fix isn't a bigger model — it's the right division of labour. Percentile math is exactly what SigNoz's aggregation is for, so I compute the comparison in the tool and hand the model a factual, hard-to-hallucinate headline. The model's job is to orchestrate and interpret, not to do nanosecond arithmetic in its head:

headline = (f"llm.chat p95 = {llm_ms} ms; slowest tool ({tool_name}) "
            f"p95 = {tool_ms} ms; the model call is ~{ratio}x slower than the tool call.")
return {"headline": headline, "operations": ops}

With that grounding, the diagnosis became correct and specific — and you can see the exact numbers it read, captured in the tool span's result attribute:

llm.chat p95 101,375.3 ms vs the slowest real tool at 1.5 ms. The agent read that and concluded, correctly, that it is ~67,000× bottlenecked on the model — not its logic, not its tools.

The observer effect is real (and it's in the trace)

One honest wrinkle: the introspection runs are themselves slow (CPU inference), so they pollute the very data the agent reads. The first version of my latency tool proudly reported that agent.introspect was the slowest operation — because it was measuring itself. The mcp.* and telemetry-reading tool.* spans, dominated by the SigNoz round-trip, leaked into the "tool" numbers too and shrank the ratio from ~67,000× to a meaningless 12×.

The fix was to exclude the introspection scaffolding (agent.introspect, mcp.*, the three reader tools) from the comparison so the agent reasons about its real request-serving workload. But it's a genuine lesson worth stating plainly: when an agent observes itself, it perturbs what it observes. You can see it happen span by span.

Why this is a different corner of MCP

SigNoz's own MCP examples — reasonably — show a frontier cloud model (Claude, Cursor) querying a production stack. This is the opposite extreme on purpose:

	Typical MCP demo	This
Model	Cloud frontier (Claude/GPT)	`qwen2.5:3b`, CPU-only, local
Target	Someone else's production	Its own traces
Network	SaaS	Fully offline
Point	Human asks about a service	Agent debugs itself

None of it is cloud-shaped. A 3-billion-parameter model on a laptop CPU used a standards-based protocol to read its own OpenTelemetry data out of a self-hosted backend and reached a true conclusion about its own performance. That is a self-observing agent, built from open parts.

What's honest about this

The model doesn't do the percentile math and I curated its tools — that's not cheating, it's system design: the LLM orchestrates and interprets, SigNoz computes, MCP is the contract between them. What's completely real is the loop: the agent chose to look at latency, issued a real tools/call to the SigNoz MCP server, read numbers measured from its own spans, and diagnosed itself. And it did it on hardware you already own.

The takeaway from my first and second posts was "you can't own an agent you can't observe." MCP adds a sequel: once the telemetry is an agent-callable API, the agent can observe itself — and the next step, closing the loop from diagnose to act, is suddenly a very short walk.

Stack: SigNoz v0.133 (self-hosted, WSL2 + Docker Engine), SigNoz MCP server (HTTP/streamable transport, 41 tools), OpenTelemetry Python SDK, Ollama qwen2.5:3b on CPU. Introspection agent + MCP bridge + curated tools: ~200 lines of Python. Built for the WeMakeDevs × SigNoz "Agents of SigNoz" hackathon.

The Retry Tax: what a local Llama agent's silent retries actually cost — measured in self-hosted SigNoz

Gajanan Gitte — Sat, 18 Jul 2026 15:00:33 +0000

The Retry Tax

Every "AI agent" retries. The model returns malformed JSON, a guardrail rejects the answer, a tool-call schema doesn't validate, a request times out — so the agent quietly runs the model again. That retry is usually invisible. It doesn't throw. The user still gets an answer. But you paid for the inference twice: twice the tokens, twice the latency, twice the GPU-seconds. I call it the retry tax, and until you trace it, it's a silent line item on your bill.

In an earlier post I gave a small local-Llama SRE agent OpenTelemetry eyes and found one LLM call was 84.5% of a request. This post is the follow-up I actually care about: I made that agent retry once, deliberately, and measured the tax in self-hosted SigNoz. One wasted call — a response the agent generated and then threw away — turned out to be 34% of the request's wall-clock time. I only know that because the trace shows the work that got discarded.

Everything here is local and free: SigNoz v0.133, Ollama running llama3.2:1b on CPU, and ~200 lines of instrumented Python. No cloud model, no vendor dashboard.

The failure I wanted to see

Real retries have messy causes. To study the cost of a retry without the noise of why, I injected a clean, deterministic fault: drop the first completed response exactly once per request. The key word is completed — the model already ran, the tokens were already spent. Then the agent has to infer again. That's the honest shape of a retry: you don't get the tokens back.

The whole fault is four environment-driven knobs in config.py:

# EXPERIMENT_ID tags every span + metric so "control" and "chaos" cohorts are
# directly comparable in SigNoz. CHAOS_DROP_RESPONSE_ONCE drops the first
# completed response of each request, forcing exactly one retry.
EXPERIMENT_ID   = os.getenv("EXPERIMENT_ID", "")
CHAOS_DROP_ONCE = os.getenv("CHAOS_DROP_RESPONSE_ONCE", "0") == "1"
LLM_MAX_ATTEMPTS = int(os.getenv("LLM_MAX_ATTEMPTS", "1"))
RETRY_BACKOFF_MS = int(os.getenv("RETRY_BACKOFF_MS", "250"))

The agent's LLM round-trip becomes an attempt loop. Each attempt is its own llm.chat span, so a retry literally duplicates the span inside the same trace — the retry tax made visible:

def _chat(self, messages):
    max_attempts = max(config.LLM_MAX_ATTEMPTS, 2 if config.CHAOS_DROP_ONCE else 1)
    for attempt in range(1, max_attempts + 1):
        with tracer.start_as_current_span("llm.chat", kind=SpanKind.CLIENT) as span:
            span.set_attribute("llm.attempt", attempt)
            resp = self.client.chat.completions.create(...)   # inference runs here
            in_tok  = resp.usage.prompt_tokens
            out_tok = resp.usage.completion_tokens

            # Injected fault: drop this completed response exactly once. The
            # tokens above were really generated, so we still record them
            # (status="dropped") -- that is the wasted work of the retry.
            if attempt < max_attempts and getattr(self._chaos, "armed", False):
                self._chaos.armed = False
                telemetry.record_llm(config.MODEL, in_tok, out_tok, latency_ms, "dropped")
                telemetry.record_retry(config.MODEL, "response_dropped")
                span.set_attribute("fault.injected", True)
                span.set_attribute("retry.reason", "response_dropped")
                span.add_event("retry.scheduled",
                               {"retry.reason": "response_dropped",
                                "retry.backoff_ms": backoff_ms,
                                "next.attempt": attempt + 1})
                span.set_status(Status(StatusCode.ERROR, "response dropped after completion"))
                time.sleep(backoff_ms / 1000.0)
                continue
            ...
            return choice.message

Two details make this trace useful instead of just red:

The dropped attempt still records its tokens (status="dropped"). Wasted work you don't measure isn't waste — it's a mystery.
experiment.id is stamped on every span and every metric, so a control run and a chaos run are directly comparable in the Query Builder. One counter counts the retries themselves:

def record_retry(model, reason):
    """Count one retry (e.g. a dropped response forced the agent to re-infer)."""
    _retries.add(1, _tag({"gen_ai.request.model": model, "retry.reason": reason}))

Two cohorts, same ten questions

I ran the identical load twice — ten SRE questions like "Is checkout healthy?" — and tagged them:

# control: no chaos
EXPERIMENT_ID=rt-control python run_load.py

# chaos: drop the first response of every request, once
EXPERIMENT_ID=rt-chaos CHAOS_DROP_RESPONSE_ONCE=1 python run_load.py

Then I asked ClickHouse (SigNoz's store) what the tax was:

Metric (10 requests each)	`rt-control`	`rt-chaos`	Δ
`llm.chat` spans	20	30	+50%
Dropped (wasted) calls	0	10	+10
Total tokens	7,322	11,515	+57%
— of which wasted	0	4,393	38% of all tokens
Cost (USD)	$0.000734	$0.001153	+57%
Avg latency / request	43.1 s	57.0 s	+32%
p95 latency	91.2 s	106.1 s	+16%

One retry per request — the mildest possible retry policy — cost 57% more tokens and money and made the agent a third slower. 38% of every token the chaos cohort burned went to answers it threw in the bin. Now imagine a real policy of "retry up to 3× on invalid JSON" against a paid API.

Reading the tax in a single trace

Here's one chaos request. The waterfall tells the whole story at a glance — 5 spans, 1 error:

Top to bottom: agent.invoke → a red llm.chat (the dropped attempt) → a second llm.chat (the retry) → tool.get_service_health → a final llm.chat. Two back-to-back inference spans before any useful work happens. In the clean cohort that first red span simply doesn't exist:

Click the red span and the attributes make the waste concrete — and queryable:

That single dropped call was 11.12s — 34.08% of total execution time, spent 431 tokens (413 in + 18 out), and carries a retry.scheduled span event with retry.backoff_ms: 250 and next.attempt: 2. Because these are real span attributes, "show me every dropped inference and how many tokens it wasted" is just a filter: name = 'llm.chat' AND retry.reason = 'response_dropped'.

A dashboard for the tax

Same filter, turned into a Retry Tax dashboard via the SigNoz dashboards API (v5 query-builder schema, dataSource: traces). The experiment.id tag does the control-vs-chaos split for free:

The clearest panel is Cost by Cohort: the pink rt-chaos slice ($0.00115) is over 1.5× the green rt-control slice ($0.00073) for the exact same ten questions. The value tiles count retried calls and wasted tokens straight from the span attributes — no code change, just a query. (They read a little above the clean table further up because the dashboard tallies every request in its window, warm-ups included; the table is the controlled 10-vs-10.)

Closing the loop: alert on the tax

A number on a dashboard nobody looks at is worthless. So I made the retry tax page me. SigNoz v0.133 alert rules use the new v5 queries: [] array (each query wrapped in a {type, spec} envelope) — worth knowing, because the old builderQueries map returns a flat "alert rule is not valid". The rule counts dropped attempts over a window:

{
  "alert": "Agent Retry Tax: LLM responses being retried",
  "alertType": "TRACES_BASED_ALERT",
  "version": "v5",
  "condition": {
    "compositeQuery": {
      "queries": [{
        "type": "builder_query",
        "spec": {
          "name": "A", "signal": "traces",
          "aggregations": [{ "expression": "count()" }],
          "filter": { "expression":
            "service.name = 'observable-agent' AND name = 'llm.chat' AND retry.reason = 'response_dropped'" }
        }
      }],
      "panelType": "graph", "queryType": "builder"
    },
    "op": "above", "target": 0, "matchType": "at_least_once"
  }
}

POST /api/v1/rules, and within a minute it was firing on the chaos data:

Now the moment my agent starts silently re-inferring — dropped responses, guardrail rejects, 429 back-offs — I find out from an alert instead of from a surprise invoice.

What's honest about this

I injected the fault, so don't read the exact 57% as a universal constant — your tax depends on your retry policy and how often reality trips it. But the mechanism is completely real: an LLM call that completes and then gets discarded costs you full tokens and full latency, and by default that cost is invisible because the agent recovers. Genuine sources of the same tax: JSON that won't parse, tool arguments that fail schema validation, content-filter rejections, timeouts, and rate-limit retries — each re-invokes the model, producing the same duplicated llm.chat span you see here. (This demo wires up the dropped-response path; the others share the identical signature.)

The takeaway isn't "retries are bad" — retries are how agents stay reliable. It's that an un-observed retry is a tax you can't see, can't bound, and can't alert on. Three OpenTelemetry attributes — llm.attempt, retry.reason, and the dropped-attempt's token counts — turn it into a first-class signal you can trace, chart, and page on. In SigNoz, that took one attribute loop and one filter.

If you can't observe your AI agents, you don't own them — and you definitely don't own their retry bill.

Stack: SigNoz v0.133 (self-hosted via Foundry on WSL2 + Docker Engine), OpenTelemetry Python SDK, Ollama llama3.2:1b. Agent + instrumentation + load driver: ~250 lines of Python. Built for the WeMakeDevs × SigNoz "Agents of SigNoz" hackathon.

I gave a local Llama agent OpenTelemetry eyes: tracing tokens, cost, and the latency tail in self-hosted SigNoz

Gajanan Gitte — Sat, 18 Jul 2026 15:00:28 +0000

I gave a local Llama agent OpenTelemetry eyes

"If you can't observe your AI agents, you don't own them."

That line from the Agents of SigNoz hackathon stuck with me, so I tested it literally. I took a small SRE "sidekick" agent running on a local Llama model — no OpenAI bill, no vendor dashboard — wired it up with OpenTelemetry, and pointed it at a self-hosted SigNoz. Then I watched what an agent actually does when you ask it a question.

The short version: the tool calls were never the problem. One LLM call was 84.5% of the entire request. I only know that number because the agent had eyes.

This post is the narrow, practical walkthrough: self-host SigNoz, instrument an agent with the OpenTelemetry GenAI semantic conventions, and read the trace, tokens, cost, and latency tail that come out the other side.

The stack (all local, all free)

Piece	What I used
Observability backend	SigNoz v0.133 (self-hosted via Foundry)
Telemetry	OpenTelemetry Python SDK (manual spans + metrics + logs)
Transport	OTLP/HTTP → `:4318`
LLM	Ollama running `llama3.2:1b` and `llama3.2:3b` on CPU
Agent	~200 lines of Python: a tool-calling loop

Heads-up if you're following an old tutorial: the install.sh script and the bundled docker-compose files are deprecated as of v0.130. SigNoz now installs through Foundry, a small "observability-stack-as-code" CLI. It's still three steps:

# 1. install the Foundry CLI
curl -fsSL https://signoz.io/foundry.sh | bash

# 2. a five-line casting.yaml (this is the whole file)
#   apiVersion: v1alpha1
#   kind: Installation
#   metadata: { name: signoz }
#   spec: { deployment: { flavor: compose, mode: docker } }

# 3. deploy
foundryctl cast -f casting.yaml

I'm on Windows, so I ran everything inside WSL 2 with Docker Engine — not Docker Desktop. That's not a personal quirk: SigNoz's own docs warn that ClickHouse Keeper crash-loops (exit 139, segfaults) under Docker Desktop's VM layer on Windows. I hit exactly that before switching, so consider this your warning too.

A minute later the UI is on http://localhost:8080, and its onboarding checklist lights up green the moment your app starts sending data:

Three ticks — logs, traces, and metrics ingestion active — and my service observable-agent shows up in the services list. That's the whole "did my telemetry land?" question answered in one screen.

The agent: a tiny SRE sidekick

The agent answers on-call questions like "Is checkout healthy?" or "What's my error budget?". It has four tools:

get_service_health(service)
list_recent_deploys(service)
calculate_error_budget(slo_percent, actual_error_rate)
search_runbook(topic)

It runs the classic agent loop: send the question + tool schemas to the model, let the model pick tools, run them, feed results back, and let the model write the final answer. Nothing exotic — which is exactly the point. This is the shape of most "AI agents" in production.

What makes it observable is a three-level span tree:

agent.invoke        (SERVER)   ← one per user question
└─ llm.chat         (CLIENT)   ← each model round-trip
└─ tool.<name>      (INTERNAL) ← each tool execution

Instrumenting the LLM call the "right" way

OpenTelemetry has GenAI semantic conventions — a standard set of gen_ai.* attributes so any backend can understand an LLM call. Instead of inventing my own keys, I set the standard ones on each llm.chat span:

with tracer.start_as_current_span("llm.chat", kind=SpanKind.CLIENT) as span:
    span.set_attribute("gen_ai.system", "ollama")
    span.set_attribute("gen_ai.operation.name", "chat")
    span.set_attribute("gen_ai.request.model", model)

    resp = client.chat.completions.create(model=model, messages=msgs, tools=TOOL_SCHEMAS)

    span.set_attribute("gen_ai.response.finish_reason", resp.choices[0].finish_reason)
    # one helper writes tokens + cost to BOTH the span and the metrics:
    record_llm(model, resp.usage.prompt_tokens, resp.usage.completion_tokens, latency_ms)

That record_llm helper is where the trick lives — the same numbers land on the span and on OpenTelemetry metrics in one place:

def record_llm(model, in_tok, out_tok, latency_ms):
    cost = cost_usd(model, in_tok, out_tok)
    attrs = {"gen_ai.request.model": model, "gen_ai.system": "ollama"}
    token_counter.add(in_tok,  {**attrs, "gen_ai.token.type": "input"})
    token_counter.add(out_tok, {**attrs, "gen_ai.token.type": "output"})
    cost_counter.add(cost, attrs)
    llm_latency.record(latency_ms, attrs)
    span = trace.get_current_span()                 # the live llm.chat span
    span.set_attribute("gen_ai.usage.input_tokens",  in_tok)
    span.set_attribute("gen_ai.usage.output_tokens", out_tok)
    span.set_attribute("gen_ai.usage.cost_usd",      round(cost, 6))

So one call feeds two questions: "what happened in this exact request?" (the span) and "what's happening across all of them?" (the metrics gen_ai.client.token.usage, gen_ai.client.cost, gen_ai.client.operation.duration).

That's it. No agent framework, no auto-instrumentation magic — just the standard conventions, so SigNoz knows exactly what it's looking at.

A note on the spec (I checked the docs, as the guide advises). The OpenTelemetry GenAI semantic conventions are at Development stability and recently moved to their own repo. A few honest deviations in the code above: gen_ai.system is now deprecated in favour of gen_ai.provider.name; gen_ai.usage.input_tokens/output_tokens are the current names (the prompt_tokens/completion_tokens you see are the OpenAI SDK's field names, which I map onto the span); and gen_ai.response.finish_reasons is officially plural (a string array). gen_ai.usage.total_tokens, gen_ai.usage.cost_usd, and the gen_ai.client.cost metric are my own extensions — the spec has no cost signal yet. The spec also models gen_ai.client.token.usage as a Histogram and gen_ai.client.operation.duration in seconds; I kept a counter and milliseconds for readability. And per the span guide the recommended span name is {operation} {model} (e.g. chat llama3.2:1b) — I use llm.chat because it reads cleaner in a waterfall.

Reading the trace: where did the time actually go?

Here's a single question — "Inventory feels slow. Pull its health, recent deploys, and the right runbook." — as one distributed trace in SigNoz:

Six spans, zero errors. Read top to bottom:

agent.invoke — the whole request, 1.83 min.
llm.chat (17 s) — the model reads the question and decides which tools to call.
tool.get_service_health, tool.list_recent_deploys, tool.search_runbook — three tools, each ~1 ms.
llm.chat (1.54 min) — the model synthesizes the final answer from the tool output.

The tools are a rounding error. The first LLM call is quick. The final synthesis call is the whole story — and SigNoz labels it for you when you click the span:

1.54 mins — 84.51% of total exec time. Right there in the drawer are the GenAI attributes I set: gen_ai.request.model = llama3.2:1b, input_tokens = 286, output_tokens = 120, total_tokens = 406, gen_ai.usage.cost_usd, and finish_reason = stop. SigNoz even captured the completion text as a span event.

This is the payoff of the semantic conventions: I didn't build a custom "LLM view." I set standard attributes, and the generic trace UI became an LLM debugger. The lesson — the long-context generation, not the tools, owns your p95 — is the kind of thing you'd never guess from logs alone.

From one trace to a fleet: the dashboard

One trace is a debugging tool. To own the agent I need aggregates, so I built an LLM & Agent Observability dashboard straight from the same span attributes:

Across a short load run of varied SRE questions:

20 LLM calls · 6,064 input tokens · 1,439 output tokens · $0.00075 total cost
Token usage over time, split into input vs. output — input dominates because every tool result is re-fed into the prompt.
LLM latency p95/p99 by model — the tail climbs past 1.6 min on CPU, which is precisely the cold, uncomfortable truth you want on a dashboard instead of in a user complaint.
Span latency p95 by operation — the single clearest panel on the board: agent.invoke and llm.chat ride high while every tool.* span sits pinned to the floor. The waterfall lesson, now true across the whole fleet.

About that cost: locally it's essentially zero (I compute it from a per-model price table). But the instrumentation is identical to what you'd use against a paid API. Point the provider at openai or anthropic instead of ollama and the same dashboard is now watching real dollars — before the invoice does.

My favorite SigNoz feature

The trace → span drawer won me over. Because SigNoz speaks the OpenTelemetry GenAI conventions natively, a plain span turned into a per-call LLM inspector — model, tokens, cost, finish reason, and the completion event — with zero custom UI. Add the "% of total exec time" on each span and root-causing a slow agent stops being guesswork.

Runner-up: the metrics query builder let me group the same gen_ai.* data by model and token type without writing SQL.

Honest takeaways

Instrument the standard, not the vendor. OpenTelemetry GenAI attributes made a generic backend understand my agent. No lock-in, no framework.
Agents hide their latency in one span. My tools were ~1 ms; a single synthesis call was 84.5%. You cannot fix what you cannot see.
Self-hosting SigNoz is a five-line casting.yaml and one foundryctl cast. Traces, metrics, and logs in one place, on my laptop, for free.
Local models make great observability test rigs. CPU inference is slow — which means the interesting tail exists, so you actually have something to observe.

Next, I'm turning this into a Track 01 project: a self-healing SRE sidekick that reads its own SigNoz traces to decide what to do. But that's another post.

Everything here — the agent, the OpenTelemetry wiring, and the dashboard JSON — is reproducible; the commands above are the ones I actually ran.