Tracing AI Agents with OpenTelemetry: A Design for LLM and Agentic AI Observability

title: "Tracing AI Agents with OpenTelemetry: A Design for LLM and Agentic AI Observability"
published: false
description: "A run-centric trace model for LLM and agentic AI observability with OpenTelemetry GenAI conventions, with measurements from a verified example."
tags: opentelemetry, observability, devops, bonree

LLM applications and agentic AI operations share a property that classic APM was not designed for: the execution path is decided at runtime by a model. A request that returns HTTP 200 may have looped, retried a failing tool, or consumed several times the usual number of tokens, and none of that is visible from the response.

This article argues for a run-centric trace model, implements it with the OpenTelemetry GenAI semantic conventions (gen_ai.*), and reports what the trace contains when an example agent is executed. It then covers the production trade-offs that decide whether such tracing survives real traffic: sampling, cardinality, content capture and context propagation across agents.

Conclusion. One trace per run, three span types (invoke_agent, chat, execute_tool), token usage on every model-call span and error status on every failed tool call are sufficient to answer three of the four questions that recur in agent incidents: which path was taken, what it cost, and what was changed. The fourth, whether the answer was correct, needs an evaluation signal that tracing cannot supply.

Why request-centric tracing falls short

Four properties distinguish an agent run from a request:

• The path is chosen at runtime. The agent plans, calls a tool, observes, plans again and may hand off to another agent. The shape of the trace varies from run to run.
• Cost is driven by tokens, which depend on loop count and context size, not on request count.
• Actions have side effects. A tool call can update a record or issue a refund, which a model-call span cannot reveal.
• Failure can look like success. A fast, well-formed, wrong answer produces no error and no latency anomaly.

The consequence is that the entity to observe is the run, and its internal structure matters more than its endpoint.

A run-centric trace model

Span name	gen_ai.operation.name	Represents
invoke_agent {agent}	invoke_agent	The whole run
chat {model}	chat	One model request
execute_tool {tool}	execute_tool	One tool invocation

Three design choices are worth stating explicitly.

• Model calls and tool calls are child spans, not events on the run, because they have their own duration, status and attributes, which events lack.
• Retries are events on the parent, because a retry is a decision of the parent and not an operation with its own cost.
• Anything outside the specification goes under an application-owned namespace (app.*), so that a later addition to the specification cannot collide with it.

Reference implementation

The listing is complete and runs without an API key. The model client and the tools are stubs, and the refund tool is configured to fail once and then succeed, so every run is identical.

import time

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter

provider = TracerProvider(resource=Resource.create({"service.name": "support-agent"}))
provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("support-agent")

PROVIDER = "example-provider"
MODEL = "example-model"


def call_llm(prompt: str) -> dict:
    """Stand-in for a real model client."""
    time.sleep(0.2)
    return {
        "text": "...",
        "input_tokens": len(prompt.split()) * 2,
        "output_tokens": 48,
        "finish_reason": "stop",
    }


def traced_chat(prompt: str) -> dict:
    with tracer.start_as_current_span(f"chat {MODEL}") as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.provider.name", PROVIDER)
        span.set_attribute("gen_ai.request.model", MODEL)
        resp = call_llm(prompt)
        span.set_attribute("gen_ai.usage.input_tokens", resp["input_tokens"])
        span.set_attribute("gen_ai.usage.output_tokens", resp["output_tokens"])
        span.set_attribute("gen_ai.response.finish_reasons", [resp["finish_reason"]])
        return resp


def traced_tool(name: str, fn, **args):
    # An exception raised inside the span is recorded and the span status is set
    # to ERROR automatically by the SDK; we only add the semantic-convention
    # error.type attribute so failures can be grouped by class.
    with tracer.start_as_current_span(f"execute_tool {name}") as span:
        span.set_attribute("gen_ai.operation.name", "execute_tool")
        span.set_attribute("gen_ai.tool.name", name)
        try:
            return fn(**args)
        except Exception as exc:
            span.set_attribute("error.type", type(exc).__name__)
            raise


def lookup_order(order_id: str) -> dict:
    return {"order_id": order_id, "status": "delivered"}


_transient_failures = {"issue_refund": 1}  # simulate one timeout, then success


def issue_refund(order_id: str) -> dict:
    if _transient_failures["issue_refund"] > 0:
        _transient_failures["issue_refund"] -= 1
        raise TimeoutError("upstream refund API timed out")
    return {"refunded": True}


def run_agent(question: str) -> None:
    with tracer.start_as_current_span("invoke_agent support-agent") as root:
        root.set_attribute("gen_ai.operation.name", "invoke_agent")
        root.set_attribute("gen_ai.provider.name", PROVIDER)
        root.set_attribute("gen_ai.agent.name", "support-agent")

        model_calls = 0
        traced_chat(question)
        model_calls += 1
        traced_tool("lookup_order", lookup_order, order_id="A-1042")
        traced_chat("order found, decide next step")
        model_calls += 1

        for attempt in range(1, 3):
            try:
                traced_tool("issue_refund", issue_refund, order_id="A-1042")
                break
            except TimeoutError:
                root.add_event("tool_retry", {"tool": "issue_refund", "attempt": attempt})

        traced_chat("write the final answer")
        model_calls += 1
        root.set_attribute("app.agent.model_calls", model_calls)  # custom attribute


if __name__ == "__main__":
    run_agent("Where is my order A-1042, and can I get a refund?")

Four notes on the listing:

• Errors. With default SDK behavior, an exception that propagates out of a span is recorded as an event and sets the span status to ERROR. The only addition is error.type, which makes failures groupable by class.
• Tokens stay on the span that produced them. Aggregates per run, per agent and per model can be derived later. The reverse is not possible.
• Finish reasons are recorded because a truncated or filtered output otherwise reads as success.
• Prompts and completions are not stored. The reasons are covered under production trade-offs below.

What the measurements show

Running the listing produces seven spans: one invoke_agent, three chat and three execute_tool.

Span	Status	Notes
chat example-model (1st)	unset	22 input / 48 output tokens
execute_tool lookup_order	unset
chat example-model (2nd)	unset	10 input / 48 output tokens
execute_tool issue_refund (1st)	ERROR	error.type = TimeoutError, exception event recorded
execute_tool issue_refund (2nd)	unset	The retry succeeded
chat example-model (3rd)	unset	8 input / 48 output tokens
invoke_agent support-agent	unset	tool_retry event; app.agent.model_calls = 3

Three observations follow directly from this data.

1. The run consumed 40 input and 144 output tokens, so cost per run is the sum over chat spans.
2. The run succeeded, yet the trace still records one failed refund attempt. A metric that only counts outcomes would report this run as fully healthy.
3. The retry is an event on the parent span, so retries can be counted per agent without searching logs.

In this example the model stub dominates run time by construction. In a real system, the split between model time and tool time is exactly what the trace is meant to reveal.

Production trade-offs

Sampling. Head-based sampling decides before a run starts and therefore discards the rare runs that matter: failures, loops and unusually expensive runs. Tail-based sampling (for example, the OpenTelemetry Collector's tail sampling processor) decides after the run completes. It can keep every run with an error status, a high model-call count or high token usage, plus a small fraction of ordinary runs as a baseline.

Cardinality. Identifiers such as conversation IDs or order numbers belong on spans, not on metric dimensions. Derive metrics from spans with bounded labels: agent name, model, tool name and error.type. The GenAI conventions also define client metrics for token usage and operation duration, which fit this pattern.

Content capture. Prompts and completions are large and often sensitive. Keep capture off by default, enable it per environment behind redaction, and prefer storing a reference to the payload over the payload itself.

Context propagation across agents. When one agent hands work to another over HTTP or a message queue, propagate the W3C trace context in the request headers or message metadata. Otherwise the run fragments into disconnected traces, and the seam, where multi-agent failures tend to occur, becomes the part that cannot be seen.

Specification drift. Parts of the GenAI conventions are still experimental. Keep attribute names in one module so that a rename is a one-line change, and pin the instrumentation version.

From tracing to agentic AI operations

Agents that operate production systems, for example to perform root cause analysis, are both consumers and subjects of observability data.

As consumers, their conclusions should reference the telemetry they rely on (topology, metrics, errors), so that an engineer can verify a diagnosis instead of repeating it. As subjects, their own runs need the trace structure above, plus a record of which actions were proposed, approved and executed. Determinism cannot be demanded of the model, but it can be demanded of the system around it: pinned model and prompt versions, typed tools with narrow permissions, and replayable runs.

Bonree ONE's AI Observability applies this model to applications built on LangChain, LangGraph, Dify and OpenClaw, and places the agent data next to the services beneath it. Bonree ONE Sage AI, Bonree's agent operations platform, builds its fault diagnosis on topology, metric and error data of the kind described here.

Limits

• Tracing records behavior, not correctness. A run can be fast, error-free and wrong, so an evaluation signal (offline test set, online judge or human review) has to be attached to the run.
• The example uses stubs, so it shows structure, not realistic latency distributions or token counts.
• The GenAI conventions are still evolving, and attribute names may change.

Conclusions

• Model an agent run as a tree: invoke_agent with chat and execute_tool children.
• Record tokens, finish reasons, tool names and error status on every span.
• Use tail-based sampling so that failures and expensive runs are always kept.
• Keep identifiers on spans and bounded labels on metrics.
• Propagate trace context across agent hand-offs.
• Add an evaluation signal, because latency and errors cannot establish correctness.

A broader checklist for reviewing agent instrumentation is available at agent-trace-checklist.

Disclosure: I am part of the Bonree team. The code above works with any OTLP-compatible backend.