Agent observability: How to trace prompts, tools, and models in production?

#ai #agents #llmops #aiops

TL;DR: Agent observability is the practice of tracing everything a multi-step AI agent does: the prompts it sends, the tools it calls, the models it routes to, and what each step costs and how long it takes. A single agent request can fan out into dozens of model and tool calls, so per-request tracing is the only way to debug failures and control spend. TrueFoundry's AI Gateway captures this across LLM and MCP traffic in one dashboard.

When an agent misbehaves in production, the hard part is not knowing that it failed. It is knowing where. A single user request can trigger several model calls, a handful of tool invocations through MCP, a routing decision or two, and a guardrail check, all before an answer comes back. Without agent observability, all you see is a slow or wrong response and no trail explaining it.

This guide covers what agent observability actually means, what you need to instrument, and how TrueFoundry's AI Gateway captures it without you wiring up a tracing stack by hand.

What is agent observability?

Agent observability is the ability to see, measure, and debug the full path of an agent request as it moves through models, tools, and policies. Classic application monitoring tells you a service returned a 500. Agent observability tells you that the agent called the search_tickets tool, got a timeout, retried on a fallback model, hit a rate limit, and finally answered from a degraded context.

The reason it needs its own name is that agents are not single API calls. They are loops. The model decides what to do, calls a tool, reads the result, and decides again. Each hop is a place where latency, cost, and correctness can drift, and each hop needs to be recorded.

Agent observability matters most when:

Your agents call tools through MCP servers and you need to know which tool is slow or failing.
You route across multiple models or providers and want to see where traffic actually goes.
You are trying to attribute cost to a team, application, or tenant.
You run guardrails and need proof of what was blocked, flagged, or mutated.

What to instrument in an agent

Good agent observability comes down to capturing the right signals at every hop. The ones that matter in practice:

Requests and throughput. Requests per second across LLM calls, MCP tool calls, and agent responses, so you can see load and spot spikes.
Latency, broken down. Not just an average. End-to-end request latency plus streaming-specific numbers like time to first token (TTFT), inter-token latency (ITL), and time per output token (TPOT), at P50, P90, and P99.
Failures by type. Failure rates split by HTTP status and error type, per model and per tool, so a flaky MCP server does not hide inside an aggregate.
Cost and tokens. Input and output tokens and dollar cost over time, attributable by model, user, team, or custom metadata.
Tool-level detail. Which MCP tools get called most, which are slow, and which error out. Server-level averages are not enough when one tool is the problem.
Policy outcomes. Guardrail results (allowed, blocked, flagged, mutated) and routing or rate-limit decisions, so governance is auditable.

If you only track top-line latency and cost, you will see that something is wrong but not why. The tool and model breakdowns are where debugging actually happens.

How TrueFoundry handles agent observability

TrueFoundry's AI Gateway is OpenTelemetry-compliant and traces every request from prompt to tool and model execution, so it plugs into Grafana, Datadog, or Prometheus if you already run them. It also ships a built-in Metrics Dashboard that covers the signals above across both LLM and MCP traffic. A few things it gives you without custom instrumentation:

One view over LLM and MCP. The overview tab shows total cost, total LLM calls, and total MCP calls together, with incoming requests broken down by endpoint (/chat/completions, /mcp-server, /agent/responses, and more).
Model metrics with real latency percentiles. Requests per second, failure rate by error type, and latency including TTFT, ITL, and TPOT at P50 through P99, pivotable by model, user, team, virtual account, or custom metadata.
Tool-level MCP metrics. A dedicated MCP metrics view drills from server down to individual tool, showing per-tool request rate, latency, and failure breakdown, plus a method-calls breakdown (tools/list, tools/call, and so on).
Guardrail and routing visibility. Guardrail evaluations by outcome, and routing metrics showing which rules fire, how often rate and budget limits are hit, and where traffic gets load-balanced.
Export and API access. Download aggregated metrics as CSV or pull them programmatically, including a dedicated agent-metrics endpoint.

On performance, the gateway is built to stay out of the hot path: it adds roughly 3 to 4 ms of overhead and handles 350+ RPS on a single vCPU, while unifying access to 1,000+ LLMs through one OpenAI-compatible API. You get the tracing without the gateway becoming the thing you have to debug.

Best practices for agent observability

Trace per request, not per service. Tie every model and tool call back to the originating agent request so you can reconstruct the full path.
Attribute cost early. Send team, tenant, or feature as metadata from day one, so chargeback and budget questions have an answer later.
Watch tool-level failure rates. A single slow or erroring MCP tool often explains a whole class of agent failures.
Alert on percentiles, not averages. P99 latency is what your users feel when an agent stalls mid-loop.
Keep guardrail outcomes auditable. Log what was blocked or mutated, because "the agent said something it should not have" is a question you will eventually be asked.