Shipping an AI agent to production is the easy part these days. Knowing what it actually did after that, and proving it, is where most teams get stuck. Traditional monitoring tells you a service returned a 200 or a 500. It doesn't tell you whether an agent called the wrong tool, hallucinated a policy exception, or leaked a field it shouldn't have touched. Agents fail in ways that look like success: syntactically valid, semantically wrong.
This post walks through what production-grade agent monitoring actually requires, both for performance and for compliance, and includes some structural patterns you can adapt directly.
Why Standard APM Isn't Enough
A normal request/response service is deterministic. Same input, same output, predictable failure modes. Agents aren't. The same prompt can trigger a different sequence of tool calls across two runs, and an output can be well-formed and still completely wrong.
That means the unit you need to observe isn't the request. It's the reasoning chain: every tool considered, every tool actually invoked, the arguments passed, the response returned, the tokens spent, and the latency at each hop, stitched together as one traceable structure.
Run
└── Trace (one agent invocation)
├── Span: reasoning step
├── Span: tool_call (get_customer_record)
├── Span: tool_call (check_policy_engine)
├── Span: model_call (final response generation)
└── Span: guardrail_check
A span is a single unit of work (one LLM call, one tool call). A trace is the full tree of spans for one invocation. Traces roll up into threads (a conversation) and threads roll up into sessions (a user's full activity). You debug top-down, but you usually find the failure bottom-up.
The Core Metrics Worth Instrumenting
If you're setting this up from scratch, these are the signals that actually matter, split by what they're protecting.
Performance signals
- Token usage per span (not just per request, this tells you which step is eating your context window)
- Tool call latency, per tool, not aggregated
- Reasoning step count per trace
- Model version per span (critical once you're running multiple model versions concurrently)
Compliance and safety signals
- Guardrail trigger events, and which policy fired
- PII or sensitive-field access per tool call
- Output quality scores from automated evaluation ("LLM-as-a-judge") against sampled production traces
- Audit trail linking every action back to a user_id and workflow_id
A minimal span schema, expressed as a rough JSON shape, might look like this:
json
{
"trace_id": "a8f1...",
"span_id": "b2c9...",
"span_type": "tool_call",
"tool_name": "check_policy_engine",
"input": { "customer_id": "masked" },
"output_summary": "eligible=true, exceptions=0",
"tokens": { "prompt": 412, "completion": 88 },
"latency_ms": 340,
"guardrail_status": "passed",
"user_id": "u_4471",
"workflow_id": "wf_claims_review"
}
Note the masked field. If you're logging full request payloads for a compliance-sensitive workflow, you're creating a second compliance problem while solving the first.
Adopt OpenTelemetry GenAI Conventions Where You Can
You don't strictly need a standard to get this working, but building on one saves you from re-inventing span schemas per team and locks you out of vendor lock-in later. OpenTelemetry's GenAI semantic conventions are becoming the common language here, defining agent, workflow, tool, and model span types along with required latency and token-usage attributes.
Two caveats worth knowing before you build on it:
As of the current spec version, most gen_ai.* attributes still carry a "Development" stability badge, meaning attribute names can change without a major version bump. Pin your instrumentation version.
OTel gives you the transport and schema layer. It doesn't give you evaluation, alerting, or a trace-replay UI. You'll still want an agent-native observability layer on top.
Monitoring vs. Observability vs. Governance: Not the Same Layer
These three terms get used interchangeably and shouldn't be.
A dashboard that only shows uptime will miss an agent that's technically "healthy" while quietly approving loan applications outside policy. You need all three layers, and critically, the compliance layer needs to be queryable independently of the performance layer, because auditors and engineers are asking different questions of the same data.
A Practical Alerting Structure
Rather than alerting on raw thresholds alone, tie alerts to the failure mode they represent:
IF guardrail_status == "failed"
AND workflow_id in [regulated_workflows]
→ page compliance channel immediately
IF tool_call latency p95 > baseline * 2
FOR 5 consecutive minutes
→ page on-call engineer
IF eval_score < threshold
ON sampled production traces (rolling 1hr window)
→ flag for human review, do not auto-page
The distinction between "page immediately" and "flag for review" matters a lot in practice. Guardrail failures on regulated workflows are a compliance event and need a human now. A dip in eval score on sampled traces is a quality signal that needs investigation, not necessarily a 2am wake-up call.
Don't Skip the Kill Switch
Observability tells you something went wrong. It doesn't stop it from continuing to go wrong. For any agent operating in a regulated or high-stakes workflow, production monitoring needs to be paired with an operational off-switch, not just a report generated after the fact. If your monitoring stack can detect a policy violation but your team's only remediation path is "open a ticket," you don't have production-grade governance yet you have production-grade logging.
A Minimal Checklist Before You Call an Agent "Production Ready"
[ ] Every tool call emits a span with tokens, latency, and status
[ ] Traces are queryable end-to-end, not just per-request
[ ] PII/sensitive fields are masked at the logging layer, not after
[ ] Guardrail triggers are logged with the specific policy that fired
[ ] Alerts distinguish compliance events from performance dips
[ ] There's a tested, working kill switch for at least the highest-risk workflows
[ ] Sampled production traces feed back into your eval suite regularly
If more than a couple of these are unchecked, the agent might be functionally working while still being unmonitorable in any way that would hold up to an audit.
Closing Thought
The failure mode to design against isn't "the agent crashed." It's "the agent did something confidently wrong and nobody noticed for three weeks." Performance monitoring and compliance monitoring solve different problems, but they need to sit on the same trace data, because by the time you're debugging an incident, you don't want to be reconciling two different logging systems to figure out what actually happened.
Top comments (0)