Before you ship an AI agent to production, you need to know what happens when it breaks silently, and most teams don't find out until a customer complains.
The reason is structural: your agent can return HTTP 200, your logs can show success, your tests can pass, and the agent still did nothing. No tokens consumed, no output, no action taken. You asked it for a summary and it gave you an empty string. You asked it to classify an email and it returned null. The call succeeded. The agent failed.
This is different from errors. Errors are loud, your monitoring catches them, your alerting screams, you fix them. Silent failures are quieter, and they're more expensive because they hide.
What you need is a checklist, a set of things to instrument before production that would catch the failures that actually matter. Not every metric, just the ones that predict a broken agent.
The 6 failure modes to watch for
1. Zero output tokens
An LLM call that consumed input tokens but produced zero output tokens is a red flag. The model was invoked, it ran, and it produced nothing. Maybe it hit a length limit, maybe it looped and timed out mid-generation, maybe it rejected the prompt entirely.
The check is simple: output_tokens equals 0, input_tokens is greater than 0, and status is success. If this happens once, it's probably noise. If it happens twice in a row, something is wrong.
Track this per agent, per model, per day. If your agent's zero-output rate climbs above 5% on a Wednesday, you have a problem before your customer does.
2. Output summary is blank or obviously wrong
Your agent returns text, but the text is empty, or it's a hallucinated placeholder like "[No response]" or "ERROR: could not generate output." The agent didn't fail to run, it ran and produced garbage.
Capture a short summary of the output, the first 50 to 100 characters, or a key field if your agent returns structured data. Then set an alert: if more than 2 consecutive runs produce blank or obviously-empty summaries, page someone.
This is a heuristic, not a law. But it's the difference between finding out at 3am because your customer's support queue is stuck versus finding out during your afternoon standup.
3. Latency spikes (model is timing out or looping)
An agent call that takes 60 seconds when it normally takes 2 seconds is a symptom. Usually it means one of two things: the model is in a retry loop, or it's stuck generating tokens and hit your timeout.
Track p50 and p95 latency per agent. If p95 suddenly jumps 10x, it's not noise. Something changed.
4. Token consumption is wildly higher than expected
You ship an agent that normally uses 500 tokens per run. On Tuesday, the same agent is consuming 15,000 tokens per run, same input, same queries, same model. But the token use is exploding.
This usually means the agent got stuck in a loop, it's calling the model multiple times in a single run, or it's feeding its own output back into itself, or the prompt got injected with adversarial content that forces token-heavy generation.
Track input and output tokens per run. Set a ceiling: if a single run exceeds 10x the 90th percentile of that agent's normal token use, that's a signal to investigate.
5. Success rate drops below your baseline
You know what your agent's success rate is most of the time. Maybe it's 97%, maybe it's 87%. Whatever it is, that's your baseline. If it drops to 70%, you have a problem.
Most monitoring tools measure errors, exceptions, 500s, timeouts. But an agent can have a very high "success" rate (all calls return 200) while having a low "useful" rate (half the calls produce empty output). Track both.
Success rate should be defined as calls that returned a non-empty, non-hallucinated result divided by total calls. If it drops 10 points or more day-over-day, alert.
6. Cost per run is spiking
Your agent cost $0.02 per run last week. This week it's $0.50 per run, same input distribution, same model. Something is bloating the token use.
Usually this is the loop problem again, the agent calling the model multiple times per run, using a more expensive model, or the prompt got bloated. But you won't know until you have the number.
Compute cost per run from input tokens plus output tokens times model pricing. If the per-run cost jumps 5x, that's your smoke signal.
How to instrument for these checks
You don't need a fancy platform to catch these. You need:
Capture these fields per run: status (success, error, timeout), input_tokens and output_tokens, latency_ms, a short summary of the output (first 100 characters, or null if empty), cost_usd (computed from tokens times model pricing), and timestamp.
Store them anywhere, a database, a log aggregate, a monitoring tool. Even a CSV file updated every 5 minutes will work to start.
Set four alerts: zero-output rate above 5% in the last hour, two consecutive runs with blank summaries, p95 latency above 10x baseline, and per-run cost above 5x the 90th percentile.
Check them daily, or set up a dashboard to watch them during launch week.
That's it. You don't need ML, you don't need anomaly detection, you don't need sophisticated alerting. You need a checklist and the discipline to look at it before your customer's error budget is exhausted.
Why this matters before production
Silent failures are cheap to fix early and expensive to fix late. Catch a zero-output loop before production and you tweak the prompt and redeploy. Catch it at 2am because your customer can't process their data and you're in firefighting mode.
The difference is observability. Not just knowing that the agent ran, but knowing what it did when it ran.
Run through this checklist before you ship. Instrument these six signals. Check them the day you launch. You'll find problems you didn't know you had, and you'll sleep better knowing you'll find the next ones before they become incidents.
The agent that fails silently is not the one you instrumented, it's the one you shipped without looking.
Top comments (0)