An AI agent can produce a successful response while the system behind it has already experienced several failures. A single user request might trigger an agent invocation, multiple model calls, a database lookup, an external API request, a retry, and finally a response. If all of that appears as one application-level request, debugging becomes difficult.
The useful trace is not the one with the most telemetry. It is the one that shows how the agent reached its result. OpenTelemetry gives you the building blocks for that: spans for operations, attributes for important context, events for point-in-time state changes, and trace context propagation across services.
Start With the Agent Execution Boundary
The top-level span should represent the agent execution itself.
For example:
invoke_agent
├── model_call
├── execute_tool
│ └── HTTP request
├── model_call
└── execute_tool
└── database query
This structure matters because the agent is usually not the operation that actually fails. A model call can succeed while a tool times out. A tool can succeed while the next model call fails. A downstream API can return an error even though the agent runtime itself is healthy. OpenTelemetry supports nested spans, so these relationships can be represented directly in the trace.
The trace should therefore follow the execution hierarchy rather than treating the entire agent as a single opaque operation.
Model Calls Need Their Own Spans
Every meaningful model invocation should be distinguishable. At minimum, the trace should allow an operator to determine which model operation happened, how long it took, whether it failed, and how it relates to the surrounding agent execution.
Current OpenTelemetry GenAI conventions define operation names such as chat, invoke_agent, and execute_tool. However, the GenAI semantic conventions are still under development and have moved to a dedicated repository, so implementations should verify the convention version they are using rather than assuming an attribute name is permanently stable.
For production systems, this versioning detail matters. Telemetry schemas are part of your operational interface. Changing attribute names or meanings can make historical traces harder to compare.
Treat Tool Execution as a Real Operation
Tool calls deserve their own span.
Consider an agent that decides to call:
get_customer
The useful trace is not simply:
agent → success
It should expose something closer to:
agent
└── execute_tool: get_customer
└── HTTP GET /customers/{id}
Now an engineer can distinguish an agent decision from the downstream operation that actually consumed time or failed.
The tool span can contain low-cardinality information such as the tool name, operation type, and execution status. OpenTelemetry's semantic-convention guidance recommends using consistent operation names and attributes so telemetry can be correlated across different services and languages. Do not automatically put the entire tool input or output into the trace.
Tool arguments can contain customer information, credentials, tokens, documents, or other sensitive data. The current GenAI conventions explicitly warn that input messages, output messages, and tool-call data may contain sensitive information. A production trace should therefore capture enough information to understand the operation without turning the observability system into an accidental data store.
Connect the Tool to the Downstream Service
This is where distributed tracing becomes particularly valuable. Suppose an agent calls a weather tool. The tool then makes an HTTP request to another service.
Without propagation, the trace might end at:
execute_tool
With trace context propagated to the downstream service, the trace can continue into the actual HTTP operation.
That gives the operator a complete path:
User request
↓
Agent execution
↓
Model call
↓
Tool execution
↓
HTTP request
↓
External service
OpenTelemetry Python supports trace context propagation, and instrumented libraries can automatically create spans for supported dependencies.
This is often more useful than adding another dashboard. The trace connects the components that already exist.
Record Failures Where They Happen
A failed tool call should not disappear into a generic agent_failed status. The span representing the failed operation should contain the failure information. With Python, OpenTelemetry supports setting an error status and recording the exception on the span.
A simple tool wrapper can look like this:
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
tracer = trace.get_tracer("agent-runtime")
def execute_tool(tool_name, tool_fn, arguments):
with tracer.start_as_current_span("execute_tool") as span:
span.set_attribute("tool.name", tool_name)
try:
return tool_fn(arguments)
except Exception as exc:
span.set_status(Status(StatusCode.ERROR))
span.record_exception(exc)
raise
The important part is not the wrapper itself. It is the relationship between the error and the operation that produced it. If the tool failed because an HTTP request timed out, the trace should make that visible. If the model call failed before the tool was executed, that should also be visible.
Use Events for State Changes
Not every piece of information needs another span. OpenTelemetry events are useful for point-in-time occurrences inside an existing operation. The Python SDK supports adding events to a span.
For an agent, events can represent state transitions such as:
planning_started
tool_selected
retry_started
human_approval_required
fallback_selected
The distinction is useful:
A span represents work with a duration.
An event records something that happened during that work.
Creating a span for every small internal state change can make traces noisy without adding much diagnostic value.
Do Not Instrument Everything Equally
More telemetry does not automatically mean better observability. A trace containing every internal function call can become difficult to read. High-volume attributes can also increase storage and processing costs, while sensitive payloads create additional security concerns.
A better approach is to instrument around operational boundaries:
Agent
↓
Model
↓
Tool
↓
Downstream service
↓
Database / API / queue
Then add details only when they help explain behavior. Semantic-convention guidance also recommends treating sensitive, verbose, or expensive attributes carefully rather than making them mandatory by default.
The goal is not to reconstruct every line of execution. The goal is to answer the production question quickly:
What happened between the user's request and the final response?
A Useful Agent Trace
A production trace should make a few things obvious without requiring an engineer to correlate five different systems manually.
For a failed request, you should be able to see:
invoke_agent
│
├── chat
│
├── execute_tool: search_customer
│ └── HTTP request
│ └── ERROR: timeout
│
└── retry
└── execute_tool: search_customer
From this trace, the failure path is immediately visible. You can see that the model call succeeded, the tool was selected, the downstream request timed out, and the agent retried the operation.
That is much more useful than an alert saying only:
Agent request failed
The Production Rule
When instrumenting an AI agent with OpenTelemetry, start with the execution hierarchy rather than the telemetry volume. Capture the agent invocation, model operations, tool executions, and downstream dependencies as related spans. Use attributes for stable operational context, events for meaningful state changes, and exception recording for failures. Be deliberate about message content and tool arguments because observability data can contain sensitive information.
Most importantly, keep the trace readable. An AI agent is already a distributed execution system. Observability should make that system easier to understand, not add another layer of complexity.
The useful trace is the one that lets an engineer follow the request from decision → action → dependency → result and understand where the system actually broke.
Top comments (0)