The agent failures that hurt in production are not the ones that crash. They are the ones that return a perfectly good answer, throw no exception, pass code review — and quietly do three times the work per request. No try/except catches that. A trace does.
Project 11 of my Agentic AI from Zero series is the layer that makes an agent operable: OpenTelemetry-shaped tracing, a latency and cost dashboard built from the spans, alert rules for loops and failures, and a canary deployment that rolls itself back on the numbers.
The bug I shipped on purpose
Two versions of the same support agent, differing by one tuple entry:
STABLE = AgentVersion(required_evidence=("status", "total_cents"), max_steps=3)
CANARY = AgentVersion(required_evidence=("status", "total_cents", "refund_eta"), max_steps=3)
The upstream lookup_order tool does not return refund_eta yet. So the canary's evidence check never passes: it re-fetches the identical record, re-drafts the answer, checks again, and stops only at the step cap.
It never raises. It returns a correct, sensible reply every time. And it costs 2.2x more per request.
The exit condition is plain Python, deliberately not a model judgement, so the failure reproduces exactly on every run:
def _missing_evidence(record, version):
return [f for f in version.required_evidence if record.get(f) in (None, "")]
Hand-rolled tracing, vendor-shaped
The roadmap said LangSmith or Arize Phoenix. What those actually consume is OTLP spans carrying the OpenTelemetry GenAI semantic conventions — so that is what this emits, from about 120 lines of contextvar-nested context manager:
with tracer.span("llm.plan", kind="llm") as sp:
resp = client.chat.completions.create(...)
record_llm_usage(sp, model, resp.usage, "plan") # gen_ai.usage.* + cost.usd
Every exported span is a dict with traceId / spanId / parentSpanId / startTimeUnixNano / attributes / status. The exporter is a seam: JsonlExporter writes exactly what an OTLP/HTTP exporter would POST, so pointing it at a collector lands these traces in Phoenix or LangSmith with no edit to the agent. No account, no key, no lock-in — and the whole telemetry layer stays testable offline.
The dashboard makes it obvious before the alerts do
Twelve support tickets, 30% routed to the canary by md5(request_id) % 100 so the split is deterministic and a run is replayable. Real NVIDIA NIM calls on meta/llama-3.1-8b-instruct — 64 spans, 12 traces, 32 live model calls.
version reqs errors err rate p50 ms p95 ms $ total $ / request
-------------------------------------------------------------------------------------
v1.4-stable 8 1 12.5% 4689.9 9529.2 0.000706 0.000088
v1.5-canary 4 0 0.0% 6826.9 13909.3 0.000778 0.000194
Four canary requests spent more in total than eight stable ones. Token counts are the provider's real usage numbers; the dollars are the published reference rate applied to them.
Alerting on the shape of the loop
The rule that matters is not clever. Count identical tool signatures per trace:
sig = span.attributes["tool.signature"] # "lookup_order(order_id=ORD-4107)"
counts[(span.trace_id, sig)] += 1
# >= 3 in one request -> CRITICAL: not making progress
Four loop alerts fired, all four on the canary, none on stable — without the rule knowing anything about what the agent was trying to do.
One detail I got wrong on the first run: I measured failure rate across all requests pooled. The injected outage was 1 failure in 12 = 8.3%, silent under a 10% threshold. Split per version it is 1 of 8 on stable = 12.5% and it fires — and it is correctly not blamed on the canary. Alerting on a blended number is how a bad release survives.
The gates that did not catch it
✅ error-rate canary 0.0% vs stable 12.5% (Δ -12.5%, max +5%)
✅ latency-p95 canary p95 13909ms vs stable 9529ms (1.46x, max 1.50x)
❌ cost-per-request canary $0.000194 vs stable $0.000088 (2.20x, max 1.30x)
❌ no-critical-alerts 5 critical alert(s) on the canary: latency-slo; loop-detected
DECISION: ROLLBACK → active version is now v1.4-stable
The error-rate gate passed: by that measure the broken version was the healthier one. The p95 gate passed too, at 1.46x against a 1.50x limit — with eight baseline samples, p95 is just the slowest baseline request, and one slow cold call inflated it. A release gated on error rate alone promotes this build. The rollback came from the cost gate and the loop alert, which is the entire argument for gating on several independent signals.
What tracing cannot see
On one ticket the agent replied "the total amount is £207.08" when the record said 21050 pence — £210.50. Every span on that request was green. Normal latency, normal cost, no loop, no error, no alert.
Observability tells you about loops, latency, spend and failures. It tells you nothing about whether the answer was right. That needs a different mechanism — a rubric plus an LLM-as-judge co-gated by deterministic hard checks, which is what Project 10 built. Both layers, and neither covers for the other.
Testing the telemetry itself
If observability is the product it needs its own tests, so run.py ends with 15 assertions over the spans the run just produced: every span closed, parent links resolve, one root per trace, the JSONL export matches line for line, the cost accounting identity (sum of span cost = sum of trace cost = sum of version cost), nearest-rank percentiles, a fingerprint proving re-evaluating the same spans gives a byte-identical alert set, and that the rollback actually flipped traffic back. 15/15 pass, and the script exits non-zero if any fails.
The model plans the tool call and writes the replies. Every span, percentile, dollar, threshold and gate is deterministic Python — because the parts you have to trust at 3am should be code you can read, re-run and diff.
Full walkthrough with the captured trace tree: https://dev48v.infy.uk/agentic/project11-observability.html — code at https://github.com/dev48v/agentic-ai-from-zero
Next up, Project 12: a contribution back to an open-source agent framework.
Top comments (0)