Three weeks after a fintech client's support agent went live, ticket resolution quality had quietly dropped by a third. No errors in the logs. No crashes. Uptime dashboards were green the entire time. The agent was answering every question - just wrong, more often, in ways nobody noticed until a customer escalated to a human and asked why the bot had told them something the actual policy didn't say.
Nobody was lying about uptime. The service really was up. It just wasn't doing its job, and nothing in the monitoring stack was built to tell the difference.
Why Traditional APM Doesn't See Agents
APM was built for a world where code either throws an exception or it doesn't. Latency, error rate, uptime - these answer "is the system broken?" They were never designed to answer "is the system right?"
An agent can be fully operational - no exceptions, no timeouts - and still be making the wrong decision on every third call. The failure isn't in infrastructure. It's in reasoning, and reasoning doesn't throw exceptions when it goes wrong.
Four Failure Modes That Hide in Plain Sight
- Tool misuse — right tool, wrong arguments (or wrong tool entirely). No error. Just a wrong answer.
- Silent hallucination — agent invents a policy or fact with the same confident tone as a correct answer. Indistinguishable from correct in logs.
- Cost/latency creep — more retries, more reasoning steps as edge cases pile up. Token spend triples with no alert. Shows up only on the invoice.
- Goal drift — each step in a long chain looks reasonable individually, but by step six or seven the chain has wandered off the original goal.
None of these trip an error code. All of them require looking inside the reasoning chain, not just checking whether it completed.
What Observability Actually Requires
Three components, none of which traditional monitoring gives you by default:
Traces — one ID tying every downstream LLM call and tool call in a request together. Without it, reconstructing an incident means grepping unrelated logs and guessing at order of events.
Spans — every reasoning step and tool call logged with full input/output, latency, and token cost. Most teams log the final answer and skip this — which is exactly the part you need when something goes wrong.
Eval loops — a sample of production traffic scored continuously against a golden set (rubric, judge model, or human review), tracked over time so a quality dip triggers an alert instead of a support ticket.
trace_id: 8f2a-91c...
├─ span: reasoning step (412 tokens)
├─ span: tool_call lookup_order(order_id=8821) → shipped 07/10
├─ span: tool_call check_policy(topic=refund) → 30-day window
└─ span: final response, quality_score: 0.94
The Eval Loop Is What Makes This Work
Tracing tells you what happened. Eval loops tell you whether it was right. And every real incident that gets caught should get added back into the golden set as a permanent regression test — so the next deployment is scored against every prior failure automatically. Skip this and every fix is temporary; the same failure mode comes back the next time someone touches the prompt.
Tooling (vendor-neutral)
- LangSmith — tightest fit if you're already on LangChain/LangGraph
- Langfuse — open-source, self-hostable, framework-agnostic
- Arize AI — production ML/LLM observability with strong drift detection
- OpenTelemetry GenAI conventions — open, vendor-neutral instrumentation you can route anywhere later
The vendor matters less than the discipline: trace-level visibility + a golden set someone actually reviews.
30/60/90 Checklist
Days 1-30 — trace ID on every request · log full input/output per span, not just final answer · capture token cost/latency per span
Days 31-60 — build a 30-50 example golden set · automated scoring on sampled production traffic · alert on quality score, not just error rate
Days 61-90 — feed every real incident back into the golden set · review cost-per-resolution weekly · quarterly raw-trace review session
Bottom Line
An agent that never crashes isn't the same as an agent that works. Reasoning failures don't throw exceptions — so if you're only watching uptime and latency, you're watching the wrong thing.
Full breakdown with diagrams and more detail on the FAQ: aviasole.com/blog/ai-agent-observability-monitoring
Top comments (0)