Your API returned 200 OK.
Latency looks normal.
No exceptions were thrown.
Every dashboard is green.
And your AI agent gave the user the wrong answer.
That’s one of the problems teams discover when AI moves from a demo into production: a technically successful request doesn't necessarily mean a successful task.
In traditional applications, failures usually leave evidence. A request times out. A dependency fails. CPU spikes. An exception gets thrown.
AI systems can fail much more quietly.
They can call the wrong tool, retrieve the wrong context, route a request to the wrong agent, silently recover from a failed step, or simply produce an incorrect answer that looks completely reasonable.
As Michael Tuszynski, Principal Architect at Presidio, put it during the What Nobody Tells You About Running AI in Production panel:
“Failure doesn't show up as an error. Failure shows up as a really well-formed answer.”
That changes what we need to monitor.
200 OK isn't task success
Consider a relatively simple multi-agent workflow:
User
↓
Orchestrator
↓
Specialized agent
↓
Retrieval
↓
Tool call
↓
Model
↓
Response
Every component in that chain can technically succeed while the overall task fails.
The orchestrator might choose the wrong agent.
The retrieval step might return irrelevant context.
A tool might execute successfully with the wrong parameters.
The model might then take all of that information and produce a polished response.
From the application's perspective:
status = 200
latency = 8.4s
error = false
From the user's perspective:
task_completed = false
That second signal is the one that matters.
Trace the trajectory, not just the response
Once an AI application starts using agents and tools, a prompt and response aren't enough to explain what happened.
Suppose a user asks:
Find a provider near me who accepts my insurance and has availability this week.
The application might need to:
1. Interpret the request
2. Determine the required specialty
3. Route it to a provider agent
4. Search provider data
5. Check insurance eligibility
6. Retrieve appointment availability
7. Reconcile the results
8. Generate the response
If the answer is wrong, knowing that the final model call took 2.8 seconds doesn't help much.
You need the trajectory.
- Which agent was selected?
- What context did it receive?
- Which tools were called?
- What arguments were passed?
- Did something fail and get retried?
- Did the agent recover?
- Which model handled each step?
That's where tracing becomes useful.
Treat the complete user request as a trace and the individual actions as spans:
trace: provider_search
├── orchestrator
├── provider_agent
│ ├── retrieve_provider_data
│ ├── check_insurance
│ └── check_availability
├── response_synthesis
└── evaluation
Now, instead of trying to reproduce a bad response, you can reconstruct it.
This is also why instrumenting AI workflows with OpenTelemetry from the beginning makes sense. Models and agent frameworks will change. Your ability to understand what happened shouldn't depend on a particular framework.
Add the signal traditional monitoring doesn't have
Tracing tells you what happened.
It doesn't necessarily tell you whether the result was good.
That's where evals come in.
Kunal Pitale, an engineering leader focused on platform engineering at Atlantic Health, described this as the semantic layer missing from traditional monitoring.
A service can be available. The calls can succeed. The latency can be acceptable.
But was the answer correct?
Did the agent choose the right tool?
Did it actually complete what the user requested?
Production AI needs both operational telemetry and quality signals.
One useful approach is to maintain a set of known-good examples and run them whenever you change a model, prompt, skill, or tool.
Known-good task
↓
Run agent
↓
Evaluate result
↓
Compare with expected behavior
Then production failures can feed that evaluation set:
production failure
↓
inspect trace
↓
understand what went wrong
↓
add regression case
↓
change prompt/tool/model
↓
rerun evals
The same principle we've used for software regression testing starts applying to AI behavior.
Measure success at the task level
This also changes the metrics that matter.
Instead of stopping at:
request count
error rate
latency
token usage
you start asking:
task success rate
tool failure rate
retry rate
agent loop rate
evaluation score
tokens per successful task
cost per successful task
That last one is particularly interesting.
Imagine two agent configurations:
| Agent A | Agent B | |
|---|---|---|
| Cost/request | $0.08 | $0.12 |
| Task success | 62% | 94% |
Agent A looks cheaper if you're only looking at cost per request.
It looks very different when you ask what a successful outcome costs.
This is why cost, quality, and observability become difficult to separate in production AI systems.
What I'd instrument from day one
If I were putting an agent into production today, at minimum I'd want visibility into:
- End-to-end traces across agents, models, and tools
- Tool calls, arguments, failures, and retries
- Model and prompt versions
- Input and output tokens
- Cost by agent and task
- Evaluation scores
- Task success
Most importantly, I'd want those signals correlated.
High token usage alone might not mean anything.
High token usage combined with repeated tool calls, increasing latency, and a falling evaluation score tells you something is going wrong.
Green dashboards aren't enough anymore
Getting an AI application to produce an answer is increasingly easy.
Operating one reliably is a different problem.
The production questions become:
- What did the agent actually do?
- Why did it make that decision?
- Which tools and models were involved?
- Did it accomplish what the user asked?
- How much did that successful task cost?
- Can I reconstruct what happened when it gets something wrong?
Because eventually your AI system will be wrong.
The dangerous failure isn't always an exception.
Sometimes it's a confident, beautifully formatted answer delivered while every traditional health metric remains green.
The goal of AI observability is to make that failure visible.
We're working on this problem at OpenObserve by correlating complete AI sessions with agent and tool traces, model calls, token usage, cost, latency, and agent behavior.
If you're building agents in production, I'd be curious: what signal has been most useful for catching failures that your traditional monitoring missed?
Top comments (1)
I watched the webinar. Both Kunal and Michael had really valuable points