You shipped your agent last month. This month your LLM bill arrived 40% higher. Now you're digging through thousands of executions trying to figure out when it went sideways.
The problem isn't dramatic. There's no crash, no error log, no obvious culprit. The agent ran. It returned status 200 every time. But somewhere along the way, token consumption climbed, gradually enough that the spike never triggered an alarm because you never set one.
This is the second layer of agent reliability: cost governance. Silent failures (agents that succeed but do nothing) steal trust. Runaway costs steal money faster.
The fix isn't complex. It's a checklist, one that takes maybe 30 minutes per agent and saves you from discovering cost problems on a bill instead of in your monitoring dashboard.
Step 1: Establish a realistic baseline for each agent
Before you can spot abnormal, you need to know what normal looks like. For each agent, log these numbers over its first 50-100 production runs:
- Total tokens per execution (input + output combined)
- Cost per execution (USD)
- Latency (how long it takes to run)
Why 50-100? That's enough volume to smooth out variance, a few slow runs, a few fast ones, without waiting forever. You're not looking for precision, you're looking for a ballpark you can trust.
Example: Your customer research agent:
- Avg: 450 tokens/run, $0.015/run, 2.1s
- Range you've observed: 380-580 tokens, $0.012-$0.019/run, 1.8-2.8s
That's your baseline. Write it down.
Step 2: Define what "abnormal" means for each agent
Not all spikes are the same. Some are benign, some are warnings.
For each baseline you just logged, decide on two thresholds:
- Caution threshold — If a single run exceeds this, it's worth a glance. Example: 150% of your baseline token count. (For the research agent above: 450 x 1.5 = 675 tokens.)
- Alert threshold — If a run hits this, you want to know immediately. Example: 200% of baseline. (For the research agent: 900 tokens.)
Why two thresholds? The caution threshold gives you early signal without false alarms, and the alert threshold is your circuit breaker. You're not waiting for the credit card bill, you're catching the climb in real time.
Step 3: Log the baseline and thresholds somewhere you'll actually check
This is the part that fails most often. You calculate baselines, then forget them.
Store them in one place: a config file, a spreadsheet, a comment block in your agent code, or your monitoring dashboard's settings. The format doesn't matter as much as the fact that they're visible and reviewable.
Example entry:
Agent: customer_research_agent
Baseline: 450 tokens/run, $0.015/run
Caution: >675 tokens (150%)
Alert: >900 tokens (200%)
Last updated: 2026-08-03
Owner: alice@team.com
Step 4: Monitor the gap, your agent's cost signature is a canary
Once you have baselines, here's what changes: a spike in cost or tokens becomes a data point, not a mystery.
When a run exceeds your caution threshold, ask yourself:
- Did the input get more complex? (longer user prompt, more context?)
- Did the model switch? (you're using a cheaper model but ended up calling the expensive one?)
- Did the task scope creep? (the agent is doing more work than it was designed for?)
- Is this a one-off or a pattern? (a single expensive run vs. several in a row?)
Most of the time, there's a reason. Maybe your agent hit an edge case and made extra API calls. Maybe a prompt changed subtly and now the model is more verbose. But the point is you know about it during the month, not when the invoice lands.
Step 5: Revisit baselines quarterly
Agents change. You update prompts, add features, improve instructions. Every quarter, or whenever you ship a meaningful change to an agent, recalculate its baseline.
You don't need to be obsessive. If your baseline was 450 tokens and it's now 470, that's drift, not a red flag. If it's now 650, that's a signal: something structural shifted. Decide if that's intentional (good) or a surprise (time to debug).
Why this matters
Most agent monitoring today catches whether an agent failed. Cost baselines let you catch how it failed, and specifically the failures that don't crash, the ones that silently bleed budget.
A runaway cost loop, an agent making extra calls, or a model conversation spiraling, looks like success in your logs. Your baseline catches it. Your team notices it. You fix it.
The alternative is discovering it on a bill, after hundreds or thousands of overage executions. And by then, the damage is done and the debugging is a slog.
Start small. Pick your most expensive agent, the one running the most queries or calling the priciest model. Spend 30 minutes setting a baseline and two thresholds. Plug those into your monitoring. Then move to the next one.
Cost governance isn't a feature. It's a habit. And like all habits, it starts with one small, concrete thing you do today.
Running agents in production? Cost baselines are only one half of the story. Silent failures, agents that return success but do nothing, are the other. Both matter. Both are invisible to standard logging. If you're monitoring only for crashes, you're missing the two most common ways agents fail in production.
Top comments (0)