Your support agent just issued a $40 refund nobody asked for.
You check the logs. The refund tool says "refund sent." The model provider shows 30 prompts that minute from random users. The guardrail logged 200 cheerful allows. Every log is correct, and none of them can tell you what the agent read right before it decided to refund.
That's the core problem with agents: they choose their path at runtime, so logs that only record outcomes can't explain decisions.
In the full post I walk through:
- ๐ณ One trace ID + a tree of spans: turning three useless logs into one story you can read top to bottom
- ๐ท๏ธ Why every guardrail verdict needs its policy version: the difference between an explanation and a shrug
- ๐ Replay as a release gate: a new guardrail replayed over 50,000 real actions caught 13 bad refundsโฆ and would have wrongly blocked 30 legit ones
- ๐ The bill: privacy, redaction and trace volume
๐ Read it here: Agent Observability: Traces, Replay and the Refund Nobody Asked For
Curious: is anyone actually replaying production traffic before shipping a prompt or guardrail change? Or is it still on the roadmap next to "write better docs"? ๐
Top comments (0)