# Your AI Agent Is a Distributed System in Disguise — Instrument It with OpenTelemetry and SigNoz Before It Bankrupts You

My warm-up blog for the Agents of SigNoz hackathon by WeMakeDevs x SigNoz.

---

Here's a claim I'll defend for the rest of this post: the moment you wrapped an LLM in a loop and gave it tools, you stopped building an application and started operating a distributed system — one that is non-deterministic, fans out across network boundaries you don't control, and bills you per token for the privilege. Most teams are running these systems in production with less instrumentation than they'd put on a toy CRUD app. That gap is where the 2 a.m. incidents, the mystery invoices, and the silent quality regressions live.

I want to make the case, concretely, for why AI agents demand observability, what the standards-based way to do it looks like in 2026, and why an OpenTelemetry-native backend like SigNoz is the right place to send the data.

The failure modes traditional monitoring cannot see

A conventional service has a bounded, mostly-deterministic execution path. Your existing APM catches the things that go wrong there: a 500, a slow query, a memory leak. Now look at what an agent actually does on a single user request:

user query
  └─ LLM call #1 (planning)              → 1,900 tokens
       └─ tool: vector_search(k=8)       → 420 ms, 8 chunks
       └─ tool: sql_query()              → ret/err → retry x3
       └─ LLM call #2 (re-plan)          → 2,300 tokens
            └─ tool: http_fetch()        → 200 OK, 14 KB injected into context
            └─ LLM call #3 (synthesis)   → 3,100 tokens
  └─ response

That's three LLM calls, four tool invocations, three retries, and a 14 KB context injection, all hidden behind one chat bubble. Every one of these is an independent failure surface, and the interesting failures are precisely the ones your HTTP-status-code dashboards will never register:

• Cost runaway. A retry loop or an over-eager re-planning step doesn't throw — it just quietly 10x's your token spend. Cost is a primary failure mode for agents, not a billing footnote. A single misbehaving conversation can cost more than a thousand well-behaved ones.
• Unbounded tool-call loops. The agent decides to call a tool, doesn't like the result, calls it again, and again. No exception is raised. Latency climbs, tokens burn, and the only external symptom is a user who gave up.
• Silent quality degradation. This is the scary one. The system returns 200 OK with a fluent, confident, wrong answer. Nothing in your logs or metrics moves. You find out from a support ticket or a churned customer.
• Context and retrieval poisoning. A retrieval step pulls a bad chunk, or a tool injects untrusted text into the prompt. The model dutifully follows it. From the outside, everything looks nominal.
• Non-reproducibility. Sampling temperature and model-side non-determinism mean "just reproduce it locally" is often impossible. The trace is the reproduction — if you captured it.

The through-line: agent failures are semantic and economic, not just operational. You need visibility into tokens, cost, tool arguments, retrieval results, model versions, and answer quality — none of which your traditional stack was built to record.

OpenTelemetry, and why the GenAI semantic conventions matter

OpenTelemetry (OTel) is the vendor-neutral CNCF standard for emitting traces, metrics, and logs. The primitives map onto agents almost perfectly:

• A trace is one full request, end to end — the entire tree above is a single trace.
• A span is one node in that tree (an LLM call, a tool invocation, a retrieval), with a start time, duration, status, and arbitrary key/value attributes.
• Span links and parent/child relationships reconstruct the causal structure — which LLM call triggered which tool call.
• Metrics are the aggregates you alert on (p95 latency, tokens/min, cost/hour).
• Logs correlate to spans via trace IDs, so a log line is anchored to the exact step that emitted it.

What makes this usable for AI specifically is the OpenTelemetry GenAI semantic conventions — a standardized gen_ai.* attribute namespace. Instead of every team inventing its own field names, the ecosystem has converged (this is now the de facto 2026 baseline) on conventions like:

• gen_ai.system — the provider (openai, anthropic, …)
• gen_ai.request.model / gen_ai.response.model
• gen_ai.usage.input_tokens / gen_ai.usage.output_tokens
• gen_ai.request.temperature, gen_ai.request.max_tokens
• gen_ai.response.finish_reason
• gen_ai.operation.name — chat, tool_execution, embeddings, etc.

Standardized names are not a cosmetic nicety. They're what lets a backend render a purpose-built LLM view, lets you compute cost with a generic formula across providers, and lets you swap tools without rewriting instrumentation. Vendor-neutrality is the entire point — your data is portable by construction.

Here's manual instrumentation of one model call. In a real agent you'd lean on auto-instrumentation (OpenLLMetry, OpenInference, or framework hooks for LangChain/LangGraph), but seeing it by hand makes the model concrete:

from opentelemetry import trace
from opentelemetry.trace import SpanKind, Status, StatusCode

tracer = trace.get_tracer("agent.llm")

def chat(prompt: str, model: str = "gpt-4o"):
    with tracer.start_as_current_span(
        "chat", kind=SpanKind.CLIENT
    ) as span:
        span.set_attribute("gen_ai.system", "openai")
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("gen_ai.request.temperature", 0.2)

        try:
            resp = client.chat.completions.create(
                model=model,
                temperature=0.2,
                messages=[{"role": "user", "content": prompt}],
            )
        except Exception as e:
            span.set_status(Status(StatusCode.ERROR, str(e)))
            span.record_exception(e)
            raise

        usage = resp.usage
        span.set_attribute("gen_ai.usage.input_tokens", usage.prompt_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", usage.completion_tokens)
        span.set_attribute(
            "gen_ai.response.finish_reason",
            resp.choices[0].finish_reason,
        )
        # derive cost from a pricing table and attach it as a first-class metric
        span.set_attribute(
            "gen_ai.usage.cost_usd",
            estimate_cost(model, usage.prompt_tokens, usage.completion_tokens),
        )
        return resp

Nest your tool calls as child spans under the active span and you get the full causal tree — with tokens and cost attached to every node. Your black box now has structured, queryable telemetry flowing out of it.

Why the backend should be OpenTelemetry-native: SigNoz

OTel emits the data; you still need somewhere to store, query, visualize, and alert on it. The architectural decision that matters here is native vs. adapted.

Many observability vendors bolt an OTLP endpoint onto a proprietary data model — you ship OTel, they translate, and you inherit the impedance mismatch (dropped attributes, lossy conversions, a UI that was never designed around traces). SigNoz is OpenTelemetry-native: OTLP is the primary ingestion path, spans/metrics/logs are stored in a columnar store (ClickHouse) tuned for high-cardinality attributes — exactly what gen_ai.* data is — and traces, metrics, and logs are correlated by design rather than stitched after the fact. It's open-source, self-hostable via Docker or Kubernetes, with a managed cloud option.

For agents, that architecture pays off directly:

• High-cardinality attributes are first-class. Slice by gen_ai.request.model, by user, by tool, by finish reason, without the cardinality penalties that punish tag-based systems.
• Trace → metric → log correlation means you can jump from "cost spiked at 14:20" to the exact traces to the exact log lines in a couple of clicks.
• Dashboards and alerts over derived fields like gen_ai.usage.cost_usd and token counts let you alert on economic and semantic signals, not just infrastructure ones.

The part that reframes the whole workflow: the MCP server

In 2026 SigNoz shipped an MCP server (announcement, source). MCP (Model Context Protocol) is an open standard that exposes a tool surface to LLMs — and here it means an AI assistant can query your observability data in natural language: "show me the slowest traces in the last hour," "which model calls had the highest token cost today," even "build an incident dashboard for the checkout service."

Sit with the implication. The traditional loop is human → dashboard → eyeballs → hypothesis. With an MCP-queryable backend, an agent can drive that loop: an LLM can pull the offending trace, read the span attributes, correlate the logs, and construct the dashboard programmatically. Observability stops being a place you go to look and becomes an API your automation can reason over. SigNoz has already demoed a LangChain agent that queries its MCP server — closing the loop from "AI you observe" to "AI that observes."

The thesis I'm taking into the hackathon: close the loop

Here's where I think this goes, and it's the project I'm building for Track 01.

Today, observability stops at detection. You get an alert; a human does the diagnosis and the fix. But every ingredient for automating the diagnosis now exists: standardized gen_ai.* telemetry in SigNoz, and an MCP server that lets an agent read it. So why stop at the alert?

The architecture I'm prototyping — call it Sentinel — runs the full loop:

1. Instrument a target agent with OTel GenAI conventions → telemetry lands in SigNoz.
2. Detect the agent-specific failure classes traditional monitoring misses — cost runaways, tool-call loops, latency cliffs, and silent quality regressions via eval-scores-as-span-attributes.
3. Diagnose by dispatching an investigation agent that queries SigNoz through MCP, isolates the offending span, and reconstructs the causal chain.
4. Act — auto-generate an incident dashboard, emit a plain-English root-cause report pinned to the exact failing span, and apply or recommend a remediation (e.g., a circuit breaker on the runaway loop).

An AI that watches other AIs, and doesn't just tell you what broke — it tells you why and moves to fix it. That's the demo I want to put in front of the judges.

If you're shipping agents, start here

You don't need a hackathon as an excuse, and you don't need to boil the ocean. The highest-leverage sequence:

1. Add OpenTelemetry to your agent using the GenAI semantic conventions — auto-instrument first, then hand-instrument the spans that matter (tool calls, retrievals).
2. Attach cost and tokens as span attributes, and derive a cost_usd field so spend is a first-class, alertable signal.
3. Point OTLP at SigNoz — Docker to start, a few commands — and read exactly one real end-to-end conversation trace.
4. Attach an eval score to your answer spans so you can alert on quality drift, not just errors.

The scariest property of a black box isn't that it fails. It's that it fails quietly — and you learn about it from a user, or from your invoice, instead of from your telemetry.

Instrument the box. Then crack it open.

---

Building this for the Agents of SigNoz hackathon (July 20–26, 2026). Follow along or come talk shop in the SigNoz Slack community. More once the code starts shipping.