The Silent Loop: Why Latency and Token Count Together Catch What Status Codes Miss

You deploy an AI agent that calls a flaky API. The API fails 10% of the time. Your agent is wired to retry on failure, good practice, right?

Then something goes wrong. The API stays broken for an hour. Your agent keeps retrying. Each retry consumes tokens. The loop never errors out, it just cycles. Success status, because the agent finished (it gave up after N retries). But the cost? Climbing in a straight line while no work got done.

This is the infinite retry loop, not a crash, not a timeout, just silent token bleed. Your logs look healthy. Your error rate is zero. Your bill is bleeding.

The reason most monitoring misses this: error rates and request counts don't catch it. The agent isn't erroring, it's succeeding at retrying. What does catch it is a pattern you can measure right now: latency and token count moving together, in a way that doesn't match your normal execution.

The Math of the Pattern

When a healthy agent runs, there's a relationship between how long it takes and how many tokens it uses. Call this the latency-to-token ratio.

• A normal run: 2 seconds, 500 tokens. Ratio: 250 tokens/sec.
• Another normal run: 3 seconds, 800 tokens. Ratio: ~267 tokens/sec.
• Your baseline range: say, 200-300 tokens/sec.

Now the loop starts:

• First 30 seconds: 15,000 tokens. Ratio: 500 tokens/sec. Way above baseline.
• Next 30 seconds: another 15,000 tokens. Still climbing.

The signal is clear: latency is spiking, and token consumption is accelerating disproportionately. The agent is doing a lot of token work in a short time, characteristic of retry loops where the model is being re-invoked repeatedly against the same or similar inputs.

By contrast:

• A normal, slow run (agent thinking hard): latency high, tokens high, but the ratio stays in your normal band. The model had a complex problem; it thought longer and used more tokens. Proportional.
• A loop: latency high, tokens extremely high (disproportionate spike), ratio breaks baseline. The model is being called over and over in the same second window, burning tokens on retry cycles.

Building the Heuristic

Here's the threshold pattern builders should use:

Step 1: Establish your baseline

Run 20 normal agent executions. For each, compute:

ratio = output_tokens / (duration_ms / 1000)

Record the 50th and 95th percentile ratios. Call these p50_ratio and p95_ratio.

Example:

• p50: 250 tokens/sec
• p95: 400 tokens/sec (accounting for "thinking hard" runs)

Step 2: Set alert thresholds

• Caution zone: ratio > p95_ratio * 1.5 (e.g., > 600 tokens/sec)
• Alert zone: ratio > p95_ratio * 2.5 (e.g., > 1000 tokens/sec)

Step 3: Combine with duration

If latency is also above your normal range (e.g., > 2 standard deviations from mean), weight the alert higher. A 45-second run with 20,000 tokens (444 tokens/sec) might be fine. A 45-second run with 50,000 tokens (1111 tokens/sec) is almost certainly looping.

Step 4: Watch the trend, not the single run

A single run can be an outlier. Watch for the pattern across 3+ consecutive runs in a 5-minute window:

• If 2 or more runs breach the alert threshold, fire the alert.
• If the ratio stays elevated, the loop is ongoing.

Why This Catches Loops Before Cost Explodes

Consider the numbers:

• Normal agent: 500 tokens/run, $0.00075 per run (at typical pricing).
• Loop for 1 hour: 150 retries, ~5000 tokens per cycle, = 750,000 tokens, ~$1.13 per hour.
• Loop for 4 hours undetected: $4.50. Small. Still bad.
• Loop for a full day: $27. Big enough to be noticed in a weekly bill review. Too late.
• Loop caught at 30 minutes via latency+token spike: ~$0.56. Caught, stopped, learning drawn.

The latency-token ratio detects the loop in the first 2-3 runs, not after hours or days. You catch it while the cost is still negligible.

How to Instrument This (No Vendor Lock-in)

You don't need a specialized tool to start. Just log three numbers per execution:

{
"agent_name": "my_agent",
"duration_ms": 45000,
"output_tokens": 22500,
"model": "gpt-4o",
"status": "success"
}

Then compute the ratio in your log aggregator (Datadog, New Relic, CloudWatch, or a simple script):

ratio = output_tokens / (duration_ms / 1000)
if ratio > 1000: # or your alert threshold
send_alert(f"High token-per-second ratio: {ratio}")

If you're wrapping an LLM SDK (OpenAI, Anthropic, etc.), you already have access to output_tokens and request duration. Logging that pair takes three lines.

The Deeper Pattern

This heuristic works because infinite loops have a structural signature: they burn tokens fast relative to the time they're supposed to be thinking. A model pondering a hard question runs long and uses tokens proportionally. A retry loop cycles the model through shallow re-attempts, burning tokens in bursts.

By watching both dimensions together, you're detecting the imbalance that characterizes the loop without needing to parse logs, understand retry logic, or set up complex alerting rules.

It's not perfect, a pathological case might fool it. But it catches the 99% case: the silent loop that's invisible when you're watching latency alone, and eats budget fast when you're not watching at all.

Start logging latency + output tokens today. Compute the ratio. Set the thresholds. You'll catch the next loop before your bill does.

---

If you're already running agents in production, pull 20 recent runs and compute your p50 and p95 right now. That baseline is your first line of defense against cost creep.