AI agent observability means capturing your agent's reasoning cycles, tool calls, and token usage as metrics, traces, and logs. In this guide I build it in three layers with OpenTelemetry (OTEL), then take the same agent to production on Amazon Bedrock AgentCore.
Your AI agent is in production. A user asks it a question, and it takes thirty seconds, calls five tools, and gives an answer you can't explain. What did it actually do? Which tools did it call? How many times did it "think" before answering? If you can't answer that, you're running agents blind. Traditional monitoring won't help you here: CPU, RAM, and uptime watch the machine, not the reasoning.
In this post I make a travel-booking agent's normal behavior visible. No injected failures, no chaos experiments. A real agent doing its job, seen through four increasingly capable lenses:
- Agent metrics: what the run cost, with zero extra configuration
- OpenTelemetry traces: the path the agent took, step by step
- Custom trace attributes: your business context, on the same trace
- Production: the same visibility in Amazon CloudWatch via Amazon Bedrock AgentCore
Everything comes from a runnable sample repository: observability-for-agents-sample-for-aws. Each demo is keyed to a specific section of the Strands Agents observability documentation.
A note on the stack. The demos use Strands Agents, an open-source SDK that emits OpenTelemetry natively. Metrics, hierarchical traces, and span attributes are general agent-observability concepts. The same patterns carry over to other agent frameworks, and Strands is model-agnostic: works with any LLM provider (Amazon Bedrock, Anthropic, local models via Ollama, or others).
What agent are we observing?
All four demos instrument the same travel agent: it searches real sandbox flight fares (Duffel API), checks real weather (Open-Meteo), and books flights into a local SQLite ledger. The only thing that changes, demo to demo, is how much of the agent's internal behavior becomes visible, and where that visibility lives:
Layer 1 / What metrics do you get with zero configuration in Strands?
Every Strands agent run already carries its own metrics: reasoning cycle count, token usage, and per-tool call counts and timings, exposed through result.metrics.get_summary(). No extra install, no exporter, no setup. Every AI agent run has a shape, and that shape is captured before you configure anything.
Compare two lenses on the same run. First, traditional logging:
DEBUG | strands.tools.executors._executor | tool_use=<...name': 'search_flights'...> | streaming
DEBUG | strands.tools.executors._executor | tool_use=<...name': 'get_weather'...> | streaming
DEBUG | strands.tools.executors._executor | tool_use=<...name': 'book_flight'...> | streaming
John Doe's flight from JFK to MIA has been successfully booked ... booking reference BK-JSFPJ5 ...
Useful for "did this run". Useless for "how much did it cost". Now the built-in metrics, one method call:
result = agent("Book a one-way flight from JFK to MIA...")
print(result.metrics.get_summary())
{
"total_cycles": 3,
"total_duration_s": 5.13,
"accumulated_usage": {"inputTokens": 2520, "outputTokens": 209, "totalTokens": 2729},
"tool_usage": {
"search_flights": {"call_count": 1, "success_count": 1, "average_time_s": 0.721},
"get_weather": {"call_count": 1, "success_count": 1, "average_time_s": 1.434},
"book_flight": {"call_count": 1, "success_count": 1, "average_time_s": 0.006}
}
}
This is real output from an agent run, and every field answers a question a log line can't:
-
total_cycles: 3. An agent is not a single function call, it's a loop: the model calls a tool, thinks again with the result, calls another. Three cycles here. If this number is ever ten for a basic question, something's wrong, and now you can see it. -
accumulated_usage. 2,729 tokens for the whole booking. Notice input is roughly ten times output; that's typical for agents, because every tool result gets fed back into the model. This is the number that tells you how heavy each request really is. -
tool_usage. Three tools, three completely different performance profiles:search_flightsat 0.7 s (a real API call),get_weatherat 1.4 s (another API),book_flightat 6 milliseconds (a local write). Without this breakdown, "the agent is slow" is a mystery. With it, it's a diagnosis.
One more habit worth building from day one: the demo also queries the booking database directly, so you can cross-check what the agent said ("booked!") against what actually persisted. In this run, the agent's claim and the ground truth agreed.
Honest caveat: in Strands 1.47.0,
accumulated_metrics.latencyMsreads0for some LLM providers. It ships as aTODOin the provider streaming code (I verified this by reading the installed SDK source). Token counts and per-tool timings are accurate everywhere; treat the top-levellatencyMsas not-yet-implemented.
Layer 2 - How do you trace an AI agent with OpenTelemetry?
Metrics are a flat snapshot, traces are the path. A trace records the full hierarchy of one request: which reasoning cycle called which model invocation, which invocation triggered which tool, in what order, with timestamps. In Strands, turning on OpenTelemetry tracing is two lines:
from strands.telemetry import StrandsTelemetry
strands_telemetry = StrandsTelemetry()
strands_telemetry.setup_console_exporter() # print the span tree to stdout
# strands_telemetry.setup_otlp_exporter() # or send it to a collector (Jaeger, CloudWatch, ...)
StrandsTelemetry wires up the OpenTelemetry SDK and registers it as the global tracer provider. Every Agent(...) call after this is automatically instrumented; there is no manual span-wrapping of your own agent loop. Run the same travel query, and the console prints the documented span hierarchy:
invoke_agent Strands Agents # the whole run (top-level span)
execute_event_loop_cycle # one reasoning cycle
chat # the model invocation for that cycle
execute_tool search_flights # one span per tool call
execute_tool get_weather
execute_tool book_flight
Each span carries attributes. The invoke_agent span holds the totals (gen_ai.usage.total_tokens: 2725, gen_ai.request.model), and each execute_tool span holds that one call's gen_ai.tool.name, gen_ai.tool.call.id, tool.status, and the formatted tool result. That's enough to answer "did book_flight fail, and what did it return?" from the trace alone, without re-running anything.
And because this is standard OpenTelemetry, the console exporter is interchangeable with any OTEL backend. Want a visual UI locally? One Docker command starts Jaeger, one environment variable points the exporter at it, and the agent code doesn't change.
Layer 3 — How do you add business context to agent traces?
Out of the box, spans carry technical attributes: tool name, token counts, status. None of those answer "was this a high-value booking?". That context is yours to add, and the Strands traces guide documents two mechanisms. The demo uses both.
Static context. Agent-level trace_attributes attach metadata (session ID, user ID, tags) to every span the agent produces:
agent = Agent(
tools=[search_flights, get_weather, book_flight],
trace_attributes={"session.id": "demo-03-custom-trace-attributes"},
)
Dynamic context. A hook tags the active span at the exact moment a business rule fires. An AfterToolCallEvent callback runs right after each tool call finishes; at that moment, the currently open span is that tool's execute_tool span, so trace.get_current_span() reaches it directly:
from opentelemetry import trace
from strands.hooks import AfterToolCallEvent, HookProvider, HookRegistry
VIP_THRESHOLD = 50.0 # low on purpose, so sandbox fares cross it
class TagVipBookings(HookProvider):
def __init__(self, threshold: float):
self.threshold = threshold
def register_hooks(self, registry: HookRegistry, **kwargs) -> None:
registry.add_callback(AfterToolCallEvent, self._tag_if_vip)
def _tag_if_vip(self, event: AfterToolCallEvent) -> None:
if event.tool_use.get("name") != "book_flight":
return
amount = float(event.tool_use.get("input", {}).get("amount", 0))
span = trace.get_current_span()
span.set_attribute("business.booking_amount_usd", amount)
span.set_attribute("business.vip_booking", amount >= self.threshold)
Run the agent, find the execute_tool book_flight span, and the custom attributes sit right alongside the SDK's own:
{
"name": "execute_tool book_flight",
"attributes": {
"gen_ai.tool.name": "book_flight",
"gen_ai.tool.status": "success",
"business.booking_amount_usd": 88.73,
"business.vip_booking": true
}
}
The detail that matters: this lives on the trace, not in the conversation. The model never sees it. Trace attributes are OpenTelemetry span metadata, entirely separate from the message list, so they add exactly zero tokens to the agent's context. But six months from now, "show me every VIP booking this quarter" is a search on your traces.
Production — where does agent observability live when you deploy?
Everything so far lived in your terminal. That's fine while you're developing, but your agent isn't going to run in your terminal, and you won't be there watching console output. The payoff of building on an open standard: everything we made (metrics, traces, attributes) is OpenTelemetry data, and OTEL data is portable. Swap the exporter, and the agent code doesn't change.
Demo 04 deploys the same travel agent to Amazon Bedrock AgentCore Runtime. The production architecture:
- The agent runs on AgentCore Runtime (the code change is one decorator:
@app.entrypoint). - The three tools become AWS Lambda functions served through an AgentCore Gateway (a Model Context Protocol endpoint with IAM auth).
-
book_flightwrites to Amazon DynamoDB instead of SQLite. Same tool, same booking, real storage. - One added dependency,
aws-opentelemetry-distro(the AWS Distro for OpenTelemetry), ships the OTEL data to CloudWatch. The Runtime runs your agent under its auto-instrumentation automatically. - One-time account setup: turn on CloudWatch Transaction Search. Without it, traces don't appear in the console (official guide).
After invoking the deployed agent, open CloudWatch GenAI Observability and you get three views:
- Agents View: every AgentCore agent in your account, with invocations, latency, and error rates.
-
Sessions View: every conversation. Remember the
session.idfrom Layer 3? This is where it pays off: it's how you go from "something went wrong" to "here's the exact conversation". -
Traces View: the same span tree you learned to read in your terminal (
invoke_agent→ cycles →chat+execute_tool), now rendered as a visual timeline, with every attribute searchable, includingbusiness.vip_booking.
The repo ships the deployment two ways: an AWS CDK stack (cdk deploy, and cdk destroy tears down everything, DynamoDB table included) and a step-by-step boto3 notebook if you want to see every API call.
Frequently asked questions
What's the difference between logs, metrics, and traces for an AI agent?
Logs are timestamped text records of what happened ("tool X was called"). Metrics are measurements of those events (how many times, how long, how many tokens). Traces are the hierarchical timeline connecting them. A log tells you that something happened, a metric tells you how much it cost, a trace shows you the path.
Do I need OpenTelemetry for basic agent metrics?
No. In Strands, result.metrics.get_summary() is part of the base SDK: no [otel] extra, no exporter, no collector. OpenTelemetry comes in when you want traces (Layer 2 onward).
Do I need a collector to see traces?
No. setup_console_exporter() prints the full span tree to your terminal. Use setup_otlp_exporter() when you want a real backend: Jaeger locally, or CloudWatch in production.
Do custom trace attributes cost extra tokens?
No. They're OpenTelemetry span metadata, entirely separate from the message list the model sees. The model never reads them.
Does this only work with Strands Agents or AWS?
No. An agent loop, hooks, metrics, and OpenTelemetry tracing are general agent-observability concepts. The demos use Strands because these primitives are built in, and Strands is model-agnostic: works with any LLM provider with no change to the agent code. The same patterns carry over to other agent frameworks.
How does Strands' built-in observability compare to manual instrumentation?
Strands emits OpenTelemetry spans natively with no manual wrapping. In frameworks without native OTEL support, you'd instrument each tool call and reasoning cycle yourself using the OpenTelemetry SDK directly. The data structure is identical — only the setup differs.
Wrap-up: three layers, one standard
Agent observability, as built here, is three layers:
- Metrics tell you what your agent did and how efficiently: cycles, tokens, tool timings. Free with the SDK.
- Traces show you the path it took: every decision, in order, with full context. Two lines to turn on.
- Trace attributes add your context to that path, so you can search it by what matters to your business. A dictionary and a hook.
You build all three once, they travel on OpenTelemetry, and a managed runtime takes them to production with minimal configuration.
One deliberate boundary: this post is about observability, seeing what an agent already does. It is not about resilience or chaos testing (injecting failures and recovering from them); that's a different, related story. And once you can see what your agent does, the natural next step is to validate it. Evaluation builds on exactly this data. You can't validate what you can't see.
Try it yourself
The travel agent, all four demos (each self-contained, with a script and a Jupyter notebook), and both production deployment paths are in the sample repository:
→ observability-for-agents-sample-for-aws
You need Python 3.10+, uv, an API key for your LLM provider (the demos support multiple providers), and a free Duffel sandbox token. Demo 01 runs in under a minute:
git clone https://github.com/elizabethfuentes12/observability-for-agents-sample-for-aws.git
cd observability-for-agents-sample-for-aws/01-agent-metrics
uv venv && source .venv/bin/activate
uv pip install -r requirements.txt
cp .env.example .env # fill in your LLM provider API key and DUFFEL_API_KEY
uv run python test_agent_metrics.py
Clone it, run it, and stop running your agents blind. Which of your agents would surprise you most if you could see every cycle? Tell me in the comments.
References: Strands Agents observability docs · OpenTelemetry · AgentCore Observability · CloudWatch GenAI Observability
Gracias!
🇻🇪 Dev.to Linkedin GitHub Twitter Instagram Youtube
Top comments (0)