Your Agent's Bill Jumped 40%. It Never Errored Once.

You ship an AI agent to production. It runs smoothly for a week. Then one morning your LLM bill is 40% higher than expected—and you have no idea why. By the time you dig into logs, the damage is done.

The problem isn't that your agent broke loudly. It's that cost spikes are silent. Your agent succeeded. It returned tokens. It burned budget. And without a baseline to compare against, you won't see the spike until the bill arrives.

Why Baselines Matter (and Why Most Setups Skip Them)

Every LLM agent has a natural token consumption pattern. For a given input, it tends to use roughly the same number of tokens each time—within a predictable range. That range is your baseline.

When an agent deviates sharply from its baseline, something changed:

A retry loop is running (hitting rate limits, trying again, trying again).
The prompt got longer accidentally.
A tool call is looping—each iteration calling the model again.
The model switched to a more expensive variant.

The catch: a 30% spike in tokens is real and expensive, but it's invisible if you're only looking at raw counts. You need a baseline to know it's an anomaly at all.

How to Build a Token Baseline (Three Metrics)

The cleanest approach uses percentiles. Here's why percentiles work better than simple averages:

Median (p50): Half your runs use fewer tokens than this; half use more. It's stable and unaffected by outliers. If your agent usually takes 150 tokens per run and the p50 is 160, that's your normal.

95th percentile (p95): 95% of your runs fall below this threshold. It captures the "busy day" for your agent—the legitimate high-end of normal. If your p95 is 400 tokens, you expect some runs to hit that; they're not anomalies.

99th percentile (p99): The rare, expensive run. Useful for budgeting headroom, but too loose for early anomaly detection.

Why percentiles over averages? One runaway loop pushes an average up by 30%. The p95 barely moves—it's already accounting for variability. Averages lie when there are outliers. Percentiles don't.

The Signal-to-Noise Problem: When Do You Alert?

Here's where most baselines fail: they're too trigger-happy. Alert on every spike above p95, and you'll get alerts when the agent just had a legitimately complex input. Ignore real spikes, and you miss the runaway loop that costs $500.

The trick is meaningful deviation: a spike that's both large and sustained.

Single-run spike above p99? Probably legitimate—the agent got a complex query, used more tokens, moved on. No alert needed yet.

Three runs in a row above p95, with at least one above p99? Now we're talking. Multiple high-cost runs suggest something systematic changed, not a one-off.

Weekly spend 50% above last week's baseline? That's a signal worth investigating.

The math is straightforward: if your agent normally costs $0.02 per run (p50 at 100 tokens * $0.00015/token for GPT-4o) and you see three runs at $0.08 each, that's 4x normal. Investigate.

How Baseline Detection Works in Practice

Here's a concrete example. Say your agent summarizes customer feedback:

Run 1: 145 tokens
Run 2: 152 tokens
Run 3: 148 tokens
Run 4: 5,200 tokens ← anomaly
Run 5: 4,800 tokens ← anomaly again
Run 6: 156 tokens (back to normal)

Your p50 baseline: ~150 tokens. Your p95: ~160 tokens.

Runs 4 and 5 are 30–40x your baseline. One spike could be noise. Two in a row is a pattern. If you're running this agent 100 times a day, 30 of those runs suddenly costing 5000 tokens changes your bill overnight.

A baseline system flags this: "Runs 4–5 exceeded p99 + sustained above p95. Investigate: possible retry loop or tool recursion."

You check the logs, find the agent is calling a downstream API that's slow, triggering a retry loop. You add a timeout or a backoff. Crisis averted.

What You Need to Implement This

To build and track baselines, you need:

Per-agent token counts from every run. Not just successes—failed runs and retries matter too, because they're where costs spike.
Time-windowed percentiles. Recompute p50, p95, p99 weekly or daily so your baseline adapts. (If you upgraded your model, the new baseline should reflect that.)
Spike detection tied to your baseline. Runs > p99 + multiple occurrences = alert. Runs > p95 once = log, but don't alert.
Cost overlay. Baseline tokens are useful; baseline cost is what matters to your CFO. Map tokens to USD per model, then alert on cost anomalies too.

Most standard application monitoring tools (DataDog, New Relic, etc.) don't track model-specific token usage out of the box—you're usually exporting logs and computing percentiles yourself. That's labor-intensive and easy to miss.

A simpler approach: Use an observability tool built for LLM agents. The AI Agents Control Tower, for example, auto-computes p50/p95/p99 baselines per agent and alerts when a run spikes above them—no manual percentile math, no custom logging. You define your SLA (expected cost or latency), and the system flags deviations automatically.

The Baseline Mindset

The broader lesson: silent cost spikes are only silent if you have no baseline to compare against. The moment you establish what "normal" looks like for your agent—in tokens, cost, latency, or all three—anomalies become visible.

Before you ship your next AI agent: