Token budget guardrails: enforcement, not just visibility

You've been monitoring your agent's token spend for weeks. Then one run goes wild — 50x the usual token count — and by the time you notice, half your budget is gone.

Visibility is the first step. Enforcement is the second, and it's the one that actually stops the bleed.

The problem: seeing the spike after the damage

Most monitoring setups give you a dashboard. Your agent runs, consumes tokens, and at month end you see the bill. Even with a live dashboard, the response chain is slow: monitor, alert, human, decision, pause. That gap costs real money.

What you actually need is a guardrail that stops the agent from running the moment it crosses a ceiling you defined in advance. Not "alert the team" — "reject the call before it happens."

This matters because of how token explosions actually happen:

• Retry loops — an agent keeps calling the same tool because it misreads the response. Each retry burns tokens. A single run can loop 10-50 times before anyone notices.
• Context bloat — the agent accumulates conversation history or debug logs in its context window. By run 100, input tokens are 3x baseline.
• Hallucinated retries — the model thinks a tool call failed when it actually succeeded, so it tries again. Logs look clean. Cost doesn't.

In every case, the agent still succeeds — HTTP 200. The token spend doesn't.

How to set per-agent token budgets

Start with a baseline. Run your agent 20-50 times under normal conditions and record the token count per execution. This is your reference point — not average monthly spend, but typical spend per single run.

Example: a document-retrieval agent
- Run 1: 3,200 input + 450 output = 3,650 total
- Run 2: 3,100 input + 480 output = 3,580 total
- Run 3: 3,400 input + 520 output = 3,920 total
Average per run: ~3,700 tokens

From that baseline, set two thresholds:

Per-run ceiling. Set it at 2-3x typical spend. Anomalies exist — a genuinely complex query might need more — but 3x is usually where something's actually wrong. If baseline is 3,700 tokens, ceiling lands around 10,000-11,000.

Monthly budget ceiling. Divide your monthly LLM budget by expected run count, then apply a safety margin. Budget $100/month, expect 100 runs — that's $1/run. At roughly $0.002 per 1K tokens for cheaper models, that's about 500 tokens/run. Set the monthly ceiling around $80 (a 20% margin) and enforce it across all agents.

The math is simple. The enforcement is what actually matters.

Implementation: the kill-switch pattern

Once you've defined ceilings, the agent needs to check them before each call, not after.

If you're using the AI Agents Control Tower, this is built in. The kill switch is an opt-in feature that pauses your agents the moment your org crosses its monthly budget:

1. Enable the kill switch in org settings and define your monthly token/cost limit.
2. Wrap your LLM client with the SDK's enforcement layer (wrap_langchain for LangChain models) — adds a lightweight status check (~3ms, cached) before each inference call.
3. If the limit's crossed, the SDK raises OpsVeritasKilledError instead of calling the model. The agent stops. No token burn, no surprise bill.

The kill switch fails open — if the status check itself fails (a network blip), the call proceeds normally. You're protected against silent failures, not against your own infrastructure breaking.

In code:

from opsveritas import init, wrap_langchain
from langchain.chat_models import ChatOpenAI

init(api_key="your-secret")

# Wrap the model for telemetry (which agent ran, how many tokens)
model = ChatOpenAI(model="gpt-4o-mini")

# Add enforcement: blocks the call if budget is exceeded
model = wrap_langchain(model, agent_name="my_agent")

# model.invoke() now raises OpsVeritasKilledError
# if the org's monthly limit is breached

For non-LangChain setups, use the Universal Webhook to POST your agent's token telemetry to the control tower after each run. Include cost_usd in the payload and the system tracks spend in real time across all agents. The kill switch still fires at the org level, but you own the decision of whether to actually call the model.

Per-agent ceilings: a second layer

The org-wide kill switch is insurance. For finer control, add per-agent ceilings inside your agent's own execution loop:

def run_agent(query):
    baseline_tokens = 3700  # your per-run baseline
    ceiling_tokens = 11000  # 3x baseline

    response = model.invoke(query)
    total_tokens = response.usage.input_tokens + response.usage.output_tokens

    if total_tokens > ceiling_tokens:
        log.error(f"Agent exceeded token ceiling: {total_tokens} > {ceiling_tokens}")
        # retry with a simpler query, alert the user, or flag for review
        return {"status": "budget_exceeded", "data": None}

    return response

This layer catches anomalies per execution, not per month. If one run gets expensive, you know immediately, and you can retry, degrade gracefully, or fail loud.

Measuring what matters

Once guardrails are in place, track three metrics:

1. Tokens per run (p50, p95) — tells you if the baseline is drifting. A climbing trend means the agent is degrading.
2. Cost per run — token count alone doesn't account for model differences; a gpt-4o run costs differently than gpt-4o-mini.
3. % of runs hitting the ceiling — if more than 1-2% of runs exceed the threshold, either the ceiling's wrong or the agent has a real problem.

Most monitoring tools show total spend. What you need is per-execution visibility — the individual run that went wild, not just the aggregate bill.

The real lesson

Visibility without enforcement is a dashboard you check after something breaks. Enforcement without visibility is a kill switch that fires mysteriously. You need both.

Set baselines from real data. Define ceilings at 2-3x normal. Enforce before the call, not after. Track per-run metrics, not just aggregate spend. When an anomaly hits, you'll catch it in milliseconds, not at month end.

Cost governance is reliability. Silent token loops are a failure mode just as real as crashes — they just hide longer.