The Silent Failure Detection Framework: Catching Agents That "Succeed" and Do Nothing

Your LLM agent returned a response. No error, no exception. But did it actually do what you asked?

That's the silent failure problem. The system behaves normally — HTTP 200, status success — but the output is empty or nonsensical. Nothing alerts you. The customer complains first. To catch these, you need to know what makes a silent failure detectable in the first place. Here's the framework.

What you're actually hunting

A silent failure is execution that reports success but produces zero (or useless) output. The agent didn't crash — it just did nothing. The most common pattern:

• Agent calls the model.
• Model responds with ~0 output tokens, or a blank response.
• No error is raised.
• The calling code proceeds, unaware.

Two other variants show up often: an agent looping indefinitely without progress (tool calls that never advance state), and output that's structurally valid but semantically empty — "I don't know" when it should have searched.

Standard error monitoring misses all three, because there is no error. You're checking the wrong signal.

The three detection levers

1. Token counting — the baseline signal

Every LLM call produces input tokens (your prompt) and output tokens (the model's response). Measurable, deterministic, available in the response metadata from every provider.

The insight: a healthy agent call has a predictable token range, per agent. A ticket-summarizer might consume ~100-200 input tokens and produce ~50-150 output tokens, run after run. If an execution comes back with 100 input tokens and 0 output tokens, that's a signal worth flagging — not proof of failure on its own, but worth alerting on.

How to implement it: log input/output tokens for every execution, compute the median and standard deviation of output tokens over the past 30 days, flag anything below the 10th percentile or exactly zero.

baseline_median_output = median(past_30_days_output_tokens)
baseline_std = stdev(past_30_days_output_tokens)
is_anomaly = (today_output_tokens < baseline_median - 2*baseline_std) OR (today_output_tokens == 0)

2. Cost anomaly — a derived signal

Cost = input_tokens × input_rate + output_tokens × output_rate. When token consumption drops, cost drops with it. Cost is often easier to track than raw tokens because it's vendor-independent — you can aggregate across models.

If your agent normally costs $0.02-$0.05 per run and today it's $0.0001, something's very wrong. Same approach: baseline median/std over 30 days, alert below the 5th percentile. Cost lags token anomalies slightly since it's derived, but it's the easier number to reason about in business terms.

3. Output examination — the confirmatory signal

If tokens suggest an anomaly, the next question is what the model actually said. Store a short excerpt of the output (not the whole response, that's usually too large), and you can scan for empty strings, repetitive/looping output, or run a secondary AI judge against a correctness rubric.

Capture the first 500 characters, flag anything empty or suspiciously short against your baseline. Rubric-based judging is optional but valuable for high-stakes agents.

Putting it together

1. Every execution logs input_tokens, output_tokens, cost_usd, output_summary, executed_at.
2. On a schedule (nightly, or hourly at volume), compute per-agent baselines and flag anything below threshold.
3. On a flag: alert the engineer with the output summary attached so triage is immediate, and optionally auto-pause the agent if the cost anomaly is severe.

Why this works, and where it doesn't

Strengths: no false positives from error logs, since you're measuring execution quality directly rather than exceptions. Early signal — tokens are real-time, so you catch problems in minutes. Framework-agnostic — LangChain, CrewAI, raw SDK, every model call produces tokens. Low friction — logging tokens is a few lines of code.

Limitations: you need baseline history (a brand-new agent with five executions has no reliable range yet), the technique is context-dependent (agents with legitimately wide output variance need a wider baseline), and detection isn't prevention — flagging a failure doesn't stop the next expensive call, that needs an actual pause or rate limit on top.

The practical next step

1. Log tokens — add input_tokens, output_tokens, cost_usd to every execution record.
2. Compute a baseline once you have a week or two of history.
3. Set an alert on the 10th percentile of output tokens, or output_tokens == 0.
4. Triage once. When it fires, you'll either see a genuine empty response (actionable) or a legitimate edge case you exclude next round.

You won't catch every silent failure this way. You'll catch the common ones — the ones that hurt most because they're invisible until someone notices.

If you'd rather not build and maintain this plumbing yourself, that's the layer we built into the AI Agents Control Tower — token tracking, baselining, and anomaly alerting, so you define thresholds instead of infrastructure. Either way the principle holds: measure the tokens, establish the baseline, alert on the gap. Silent failures are only silent until you start listening.