DEV Community

Kunal
Kunal

Posted on • Originally published at kunalganglani.com

OpenTelemetry Instrumentation for AI Agents [2026]: Ship It

#opentelemetry #llmops #aiagents #observability

Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

OpenTelemetry Instrumentation for AI Agents [2026]: Ship It

OpenTelemetry instrumentation for AI agents is how you trace one agent run end-to-end. One trace. One story. It should include the LLM calls, retrieval-augmented generation (RAG) steps, tool calls, retries, and the final outcome. And it needs enough attributes to explain two things engineers actually care about: where the latency came from and where the money went.

Key takeaways

  • A single agent run should be one root trace, with child spans for llm, retrieval, tool, and retry/backoff so you can blame the right step for latency and spend.
  • You don’t need unstable “GenAI” semantic conventions to win. A small, forward-compatible custom schema gets you 80% of the value with 20% of the churn.
  • Token cost observability is not “nice to have”. It’s the only sane way to compute cost per successful task and quantify your agent’s error tax.
  • Tail sampling is the pragmatic move for agents: keep 100% of failures and slow/expensive runs. Downsample the boring successes.
  • Do not casually store prompts and tool arguments in traces. Redact, hash, or don’t emit them at all.

If you can’t answer “which step burned the money” from a trace waterfall, you don’t have observability. You have expensive vibes.

30-second version

AI agents look simple in a demo, then turn into a messy chain of model calls, searches, and tool runs in production. When something breaks or gets slow, you need to know exactly where the time and money went. OpenTelemetry gives you a standard way to record every step of an agent run, so you can debug failures, spot retries, and track usage costs. The trick is to keep your tracing schema small and consistent: record what happened, how long it took, whether it worked, and how much it cost. Then build a few dashboards: latency by step, success rate, and cost per successful task.

Why I’m pushing OpenTelemetry for agentic AI (and not another vendor SDK)

I don’t care how slick your agent demo is. In production, an agent is a distributed system that happens to talk.

So you get the usual distributed system nonsense:

timeouts, partial failures, queue hops, retries, “just one more tool call,” and then a fun incident where your token bill looks like someone fat-fingered an extra zero.

OpenTelemetry is the boring answer, which is exactly why it’s the right one.

  • It’s portable. Traces + metrics + logs, with an ecosystem that doesn’t disappear when a vendor changes its pricing page.
  • It matches how engineers think. A trace waterfall is legible. “Agent magic telemetry” is not.

The psychological part matters more than people admit. When an engineer can point at a trace and say, “the retrieval span took 900ms” or “this tool retried three times,” the debugging conversation stops being mystical and starts being mechanical.

One grounded lesson from building the Walmart conversational commerce chatbot (Firework/Zealsight, 2022–2024): we handled millions of queries daily with sub-second responses, and the biggest latency lever was not some clever model tweak. It was the plumbing. Event-streaming the context pipeline with Kafka mattered more than anything model-side. That’s why the trace model in this post separates retrieval/context/tool time from model time. If you don’t, you’ll waste weeks “optimizing prompts” while your retriever is the real bottleneck.

On portability: OpenTelemetry defines OTLP (OpenTelemetry Protocol) as the standard way to export telemetry. That’s the point of a common pipeline. See the OTLP Specification.

[Insert illustration: “Trace waterfall of an AI agent run showing LLM, retrieval, and tool-call spans with token cost per span.”]

What should the root trace represent in an AI agent run?

The root trace represents one user-visible unit of work. In agent land, that’s usually one of:

  • “Answer this user request” (chat)
  • “Complete this task” (background agent)
  • “Run this job” (batch)

Call it a run. Every span you emit should answer a simple question: “How did this run end up with that outcome?”

Concretely, I like:

  • Root span name: agent.run
  • Root span attributes: the “business identity” of the run
    • agent.name (which agent)
    • agent.version (so you can correlate regressions)
    • run.id (your idempotency key)
    • user.id or tenant.id (if applicable)
    • task.type (classification)
    • run.outcome (success|failure|partial|timeout|cancelled)

Two rules that keep you out of trace hell:

  1. One trace per run even if you hop processes. If your agent calls a queue worker, that worker still belongs in the same trace.
  2. Make the root span long-lived. End it only when the run is actually done. Not when the first model call comes back.

This is where trace context propagation matters. If you already do distributed tracing for microservices, this will feel familiar. If you don’t, this is still the correct mental model.

For context propagation, OpenTelemetry aligns with the W3C Trace Context standard (traceparent/tracestate). See the W3C Trace Context spec.

How to model LLM calls, retrieval, and tool calls as spans

I’m going to be annoying about this: keep your span taxonomy boring. Boring scales. Boring is debuggable.

A span taxonomy that works in real systems

Under agent.run, create spans that correspond to steps you can actually improve:

  • agent.plan (optional): planning / decomposition
  • retrieval.query (RAG): vector search, keyword search, GraphRAG traversal
  • retrieval.rerank (optional): reranker model, heuristic scoring
  • llm.generate (or llm.chat): every model completion
  • tool.call (tool use): HTTP calls, DB queries, filesystem ops, MCP tool calls
  • agent.parse / agent.validate (optional): output parsing, schema validation
  • agent.retry (or use events on the failed span): retries/backoff

If you do multi-agent orchestration, model sub-agents as:

  • agent.subrun span (child of root)
  • everything inside that sub-agent under the subrun

The whole point is that spans map to knobs you genuinely have:

  • retrieval knobs: chunking, caching, index, reranking
  • model knobs: model choice, temperature, max tokens
  • tool knobs: timeouts, concurrency limits, circuit breakers

Tool calls are not “just HTTP spans”

Yes, your HTTP client library might already emit spans. Great. Keep them.

But agent observability needs a higher-level view: “this was the payments.lookup step,” not “this random URL was slow.” The URL is trivia. The tool step is the thing your workflow depends on.

So on your tool spans, add semantic attributes like:

  • tool.name: payments.lookup, github.search, browser.fetch
  • tool.type: http|db|cache|filesystem|mcp|queue|cli
  • tool.result: success|error|timeout|rate_limited
  • tool.attempt: integer

Because when an agent run fails, you’re not asking “which endpoint?” You’re asking “which step blew up the run?”

LLM spans should be cost-attributable

Every llm.generate span should include token counts. You can compute dollars later, but you can’t compute tokens later if you didn’t record them.

Minimum set:

  • llm.provider: openai|anthropic|google|azure_openai|local
  • llm.model: e.g., gpt-4.1, claude-sonnet-4.6
  • llm.input_tokens: integer
  • llm.output_tokens: integer
  • llm.cached_tokens (if your provider reports it)
  • llm.request_id (provider request id)

If you care about streaming latency, also capture:

  • llm.ttft_ms (time-to-first-token)

This pairs well with the site’s existing focus on latency work (see LLM latency benchmarks and AI agent latency budgets).

Minimal span attribute schema (forward-compatible with evolving GenAI conventions)

OpenTelemetry semantic conventions are real. GenAI conventions are still moving.

So the move is:

  • keep a minimal set of custom attributes that you control
  • optionally map them later to whatever becomes stable

OpenTelemetry’s semantic convention registry has a “Generative AI” section (versioned). Treat it as a mapping target, not your internal source of truth. See OpenTelemetry semantic conventions.

Here’s a minimal schema I’d actually ship.

Span type Span name example Required attributes Nice-to-have attributes
Root run agent.run agent.name, agent.version, run.id, task.type, run.outcome tenant.id, user.id, run.error_class, run.success (bool)
LLM llm.generate llm.provider, llm.model, llm.input_tokens, llm.output_tokens llm.ttft_ms, llm.temperature, llm.max_tokens, llm.request_id
Retrieval retrieval.query retrieval.system (e.g., pgvector, pinecone), retrieval.top_k retrieval.query_len, retrieval.hit_count, retrieval.cache_hit
Tool tool.call tool.name, tool.type, tool.attempt, tool.result http.status_code, db.system, tool.timeout_ms
Retry agent.retry (or event) retry.count, retry.backoff_ms, retry.reason retry.policy

A few consistency notes (aka: the stuff that bites you later):

  • Use integers for token fields. Don’t stringify them.
  • Decide once whether run.outcome is an enum string or a boolean run.success. Shipping both is how dashboards become an interpretive dance.
  • Keep attribute keys stable. Changing keys is how you quietly nuke your charts.

Internal linking for context:

  • If you’re building AI agents, you’ll end up needing this schema.
  • For RAG-heavy agents, you’ll also want RAG and retrieval-augmented generation tradeoffs nailed.
  • If you’re doing relationship-aware retrieval, GraphRAG isn’t magic but it’s sometimes worth it.

Which attributes are the minimum to compute token cost and cost per successful task?

To compute cost per run, you need exactly three ingredients:

  1. Token counts on each LLM span
  2. A price map per provider+model
  3. A run-level success/failure outcome

The minimum attributes are:

  • run.id
  • run.outcome (or run.success)
  • On every llm.* span: llm.provider, llm.model, llm.input_tokens, llm.output_tokens

Everything else is nice, but not required.

Cost math (with real-looking numbers)

Assume a run has:

  • 3 LLM spans
  • 2 tool spans
  • 1 retry on a tool call

Example:

  • llm.generate #1: 1,200 input tokens, 250 output tokens
  • tool.call search_index: success, 180ms
  • llm.generate #2: 2,000 input, 400 output
  • tool.call payments.lookup: timeout, 1,000ms
  • agent.retry: backoff 300ms
  • tool.call payments.lookup: success, 220ms
  • llm.generate #3: 1,500 input, 300 output
  • run outcome: success

Now define a simple model cost function using your price map:

  • Cost per LLM span = (input_tokens * input_price_per_token) + (output_tokens * output_price_per_token)

Then:

  • Run cost = sum(cost of all LLM spans) + (optional) tool per-call cost if tools are paid APIs
  • Cost per successful task = total cost of successful runs / number of successful runs

This is where “error tax” stops being a vibe and becomes a number.

Error tax (tokens + time)

Define error tax for a run as the incremental cost/time caused by failures and retries:

  • Token error tax: tokens spent on spans that didn’t contribute to success (e.g., reprompts after tool failure)
  • Tool error tax: paid tool calls that failed
  • Latency error tax: time spent on failed spans + backoff

In the example above:

  • Tool latency error tax includes the 1,000ms timeout + 300ms backoff.
  • If the agent had to reprompt the LLM because the tool failed, those LLM tokens are part of token error tax.

I go deeper in Agent Per-Task Cost Calculation and AI Agent Cost Per Task.

As a data anchor from my own work: I run pricing tracker and cost tooling on this site, and I routinely sanity-check agent budgets against LLM cost constraints before I let a pipeline run unattended.

How do you record retries/backoff and attribute error tax to the right step?

Retries are where observability goes to die.

Teams either:

  • don’t trace them at all (so everything looks “fine”)
  • or they trace them, but they lose the relationship between attempt 1 and attempt 2, so analysis becomes guesswork

Here’s what actually works.

Option A: model each attempt as its own span

For tool.call spans, emit one span per attempt:

  • attempt 1: tool.attempt=1, tool.result=timeout
  • attempt 2: tool.attempt=2, tool.result=success

Also emit a small agent.retry span (or an event) between them:

  • retry.backoff_ms=300
  • retry.reason=timeout

This makes the waterfall obvious, which is half the battle.

Option B: one span with events (only if your UI supports it well)

Create one span tool.call payments.lookup with events:

  • event: attempt=1 result=timeout
  • event: backoff_ms=300
  • event: attempt=2 result=success

It reduces span volume, but it’s harder to analyze with derived metrics.

I strongly prefer Option A. It makes error tax computation dead simple. Group by tool.name, compute failure rate, tally retries, done.

How do you propagate trace context through async tool calls and background workers?

If your agent stays in one process, propagation is easy. The pain starts when your run crosses boundaries:

  • HTTP calls to another service
  • queue messages
  • background workers
  • cron jobs that continue a run later

The rule is simple: propagate trace context with the work.

HTTP / RPC

  • Use W3C Trace Context headers: traceparent and tracestate.
  • Most OpenTelemetry SDKs will do this for you if you use the instrumented client.

Queues and Kafka

If you use Kafka (or any message bus), put trace context in message headers. This is not optional if you want “one trace per run” to mean anything.

This is also where that Walmart lesson applies again. If your context pipeline is event-streamed (Kafka), your trace needs to show queueing and processing time. Otherwise you’ll misattribute latency to “the model” when it’s really your pipeline.

Background jobs

For background workers, you need:

  • a run.id that survives process boundaries
  • trace context passed via message headers
  • a span in the worker that’s a child of the original trace

When you get this right, “agents are distributed systems” stops being a metaphor. It’s literal.

How do you sample agent traces without losing debuggability?

Agent traces get expensive fast. Not just storage. Ingestion and query costs too.

Sampling isn’t shameful. It’s how you avoid bankrupting yourself with your own telemetry.

The sampling strategy I ship first

  1. Keep 100% of failures. Always.
  2. Keep 100% of “slow runs” above a threshold (e.g., above your p95 latency target).
  3. Keep 100% of “expensive runs” above a cost threshold.
  4. Sample successful, cheap runs at a low rate.

Those thresholds aren’t universal. They depend on your median run cost, retention requirements, trace size, and volume.

Tail sampling (why it fits agents)

Tail sampling lets you decide after you’ve seen the whole trace whether to keep it. That’s perfect for agents because you often don’t know a run is “interesting” until the end.

The OpenTelemetry Collector has a tail sampling processor in the contrib distribution (see tailsamplingprocessor in the OpenTelemetry Collector Contrib repo: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor). This is what makes “keep failures and expensive traces” practical.

Even if you don’t tail-sample on day 1, design your schema so you can add it later without tearing up all your dashboards.

Privacy: prompt and tool-argument redaction (or not storing them at all)

If you take nothing else from this post: don’t spray prompts and tool arguments into tracing backends by default.

Traces are widely accessible internally. They get pasted into tickets. They get screenshotted into Slack. If you store raw user text, API keys, or PII, you will regret it.

What I do instead:

  • Store lengths and hashes, not raw content:
    • prompt.len_chars
    • prompt.sha256
    • tool.args.len_chars
  • Redact known sensitive patterns (emails, phone numbers, credit cards)
  • Use allowlists for attributes: only emit keys you explicitly permit

If you truly need replay/debugging, use a separate secure store with tighter access controls. Don’t overload tracing for that.

For security-minded agent builders, pair this with:

Dashboards to build first: latency per step, success rate, error tax, cost per success

Traces are for debugging one run. Dashboards are how you avoid learning about regressions from your CFO.

Here are the first four I build.

1) Latency by span type (p50/p95)

Break down p50 and p95 latency for:

  • llm.generate
  • retrieval.query
  • tool.call grouped by tool.name

If p95 tool latency spikes, that’s often an upstream dependency degradation. If retrieval p95 spikes, that’s usually index/cache churn.

2) Run success rate by task type

Group by:

  • task.type
  • agent.version

If a deploy drops success rate from 98% to 92%, you want to know immediately. This is basic production hygiene for production AI.

3) Error tax dashboard

Track:

  • average retries per run
  • fraction of runs with any retry
  • cost of failed attempts (tokens + paid tools)

This dashboard changes behavior. People stop hand-waving flaky tools once the cost curve is staring them in the face.

4) Cost per successful task

Compute:

  • total tokens / successful run
  • total $ / successful run

This is where token counts on spans stop being optional.

A related data anchor from my own publishing automation: operating this site’s multi-agent blog publishing pipeline (2025–present) taught me that deterministic gates catch more failures than “just use a bigger review model.” That instinct carries into observability. Don’t wait for a human to notice cost drift. Gate and alert on it.

If you’re already thinking about budgets, read AI Agent Cost Per Task and Reduce LLM API Costs 60%.

[Insert illustration: “Dashboard showing p95 latency by span type and cost per success over time after an agent release.”]

How to connect traces to metrics/logs in an OTLP pipeline (Collector processors, exporters)

Once you emit spans, you need a pipeline. The architecture I like:

  • App emits OTLP (gRPC or HTTP)
  • OpenTelemetry Collector receives it
  • Collector processors handle:
    • batching
    • attribute transforms
    • sampling (head or tail)
    • redaction
  • Collector exports to your backend(s)

OpenTelemetry’s Collector docs are the best place to start: https://opentelemetry.io/docs/collector/

This is how you keep vendor choice flexible. As long as you speak OTLP, you can move.

One practical tip: metrics derived from spans (like “cost per run”) are worth pushing into a metrics backend for cheap long-term trend queries. Traces are for investigation. Metrics are for “is this on fire?”.

A 7-step implementation checklist (the one I wish more posts gave you)

  1. Define your root span: agent.run and what “run outcome” means.
  2. Instrument LLM spans: one span per completion, with token attributes.
  3. Instrument retrieval spans: vector search, rerank, cache hits.
  4. Instrument tool spans: explicit tool.name, tool.type, attempt/result.
  5. Model retries: per-attempt spans plus backoff span/event.
  6. Propagate context across boundaries: HTTP headers + queue headers.
  7. Ship dashboards: latency per step, success rate, error tax, cost per success.

Do these seven things and you’re ahead of most “LLMOps” teams who are still staring at one request timer and calling it a day.

Closing: my prediction for 2027

By 2027, “agent observability” won’t be a category. It’ll just be observability.

The teams that win will treat an agent run like a traceable, budgeted workflow. Not a black box model call with some vibes layered on top.

If you’re building agents today, here’s my challenge: pick one production workflow and implement open telemetry instrumentation for ai agents end-to-end this week. If you can’t explain a p95 regression or a token bill spike by looking at one trace waterfall, you’re flying blind. And the bill will teach you faster than I can.

Originally published on kunalganglani.com

Top comments (0)