Spin up 20 agents against a $50 budget. Each one checks spent < limit before calling the model, and each one passes, because none of the other 19 have reported their usage yet. A few seconds later you're at $71. Every agent behaved correctly and the budget still broke.
This is the failure I see most in homegrown cost guards. Here's the code that causes it, the code that fixes it, and a comparison of the controls people usually lean on instead.
The naive guard
spent = 0.0
LIMIT = 50.0
def call_llm(messages):
global spent
if spent >= LIMIT:
raise RuntimeError("budget exhausted")
resp = client.chat.completions.create(model=MODEL, messages=messages)
spent += cost_of(resp.usage)
return resp
It looks fine, and in a single-threaded script it mostly is. The bug is the gap between the check and the increment, which is the full round-trip to the provider and can easily run 5 to 30 seconds on a long completion. Every request that starts during that gap reads the same stale spent. Run 20 workers and you have 20 requests in flight that each believed they had the whole remaining budget.
There's a second bug as well: the check ignores how expensive the next call could be. With $0.40 left, a request that can generate 8,000 output tokens still gets through.
What the usual controls don't stop
Before writing anything, it's worth being clear about what you probably already have.
| Control | Where it runs | When it acts | What gets through |
|---|---|---|---|
| Budget alert | Billing system | After spend is recorded | Everything between the threshold and a human reading the email |
| Rate limit (RPM/TPM) | Provider | Per time window | A slow, steady loop. It limits speed, not total |
| Provider hard limit | Org/project level | Against usage the provider has tallied | One runaway agent draining the shared pool for every other user, plus overshoot while the tally catches up |
| Pre-flight reservation | Your process | Before the request leaves | Only callers that bypass the wrapper, and errors in your price table |
On alerts, the vendors are explicit. Google Cloud's budget docs say an alerts-only budget does not cap usage, and that the alert emails don't prevent billing. OpenAI's spend-limit guide says plainly that "spend alerts do not enforce a cap" and treats alerts and hard limits as separate controls.
The numbers suggest a lot of teams haven't absorbed this. A February 2026 Sapio Research survey for DoiT asked 500 finance leaders at large US and UK enterprises about AI spend: 79% had overrun their AI costs in the past year. Among organizations that called their FinOps practice "very mature or leading edge," it was 89%, with an average overspend of 30.9%. Better dashboards don't stop a bill. They just tell you about it sooner.
The fix: reserve the worst case, then settle
The pattern is the one payment systems use. Before the call, place a hold for the most the request could cost. If the hold doesn't fit, the request never gets sent. Afterwards, release the hold and record what the request actually cost.
import threading
from dataclasses import dataclass
class BudgetExceeded(Exception):
pass
@dataclass(frozen=True)
class Price:
input_per_mtok: float
output_per_mtok: float
def cost(self, input_tokens: int, output_tokens: int) -> float:
return (input_tokens * self.input_per_mtok
+ output_tokens * self.output_per_mtok) / 1_000_000
class BudgetGuard:
def __init__(self, limit_usd: float):
self.limit = limit_usd
self.spent = 0.0
self.reserved = 0.0
self._lock = threading.Lock()
def reserve(self, amount: float) -> float:
with self._lock:
committed = self.spent + self.reserved
if committed + amount > self.limit:
raise BudgetExceeded(
f"need ${amount:.4f}, have ${self.limit - committed:.4f}"
)
self.reserved += amount
return amount
def settle(self, held: float, actual: float) -> None:
with self._lock:
self.reserved -= held
self.spent += actual
Check-and-reserve happens inside one lock acquisition, so two workers can't both claim the same headroom. Ten workers reserving $5 each fill a $50 budget, and the eleventh is refused before it opens a socket.
Now wrap the client:
import openai
def estimate_input_tokens(messages) -> int:
# Deliberately pessimistic: ~3 chars/token plus per-message overhead.
# Swap in tiktoken if you want tighter holds.
chars = sum(len(m["content"]) for m in messages
if isinstance(m.get("content"), str))
return chars // 3 + 16 * len(messages)
class GuardedOpenAI:
def __init__(self, client: openai.OpenAI, guard: BudgetGuard, price: Price):
self.client, self.guard, self.price = client, guard, price
def chat(self, *, model, messages, max_completion_tokens, **kw):
worst = self.price.cost(estimate_input_tokens(messages),
max_completion_tokens)
held = self.guard.reserve(worst) # raises before any network I/O
try:
resp = self.client.chat.completions.create(
model=model,
messages=messages,
max_completion_tokens=max_completion_tokens,
**kw,
)
except openai.APIStatusError:
self.guard.settle(held, 0.0) # provider rejected it: release
raise
except Exception:
self.guard.settle(held, held) # timeout etc: assume we paid
raise
u = resp.usage
self.guard.settle(held, self.price.cost(u.prompt_tokens,
u.completion_tokens))
return resp
A few choices here are deliberate:
max_completion_tokens is required. Without a ceiling on output there's no worst case to reserve, and the guard can't work. With reasoning models this is even more important, because hidden reasoning tokens are billed as output and they count against this same limit.
Timeouts get charged at the hold. A request that timed out on your end may have finished and been billed on the provider's end. Assuming you paid costs a little headroom. Assuming you didn't is how budgets drift.
Unpriced models should fail. Build the Price from a dict lookup that raises a KeyError on unknown models. Falling back to a price of zero means the cap quietly stops working.
The Anthropic version is the same shape. max_tokens is already a required argument, and usage comes back as resp.usage.input_tokens and resp.usage.output_tokens.
Proving it under load
from concurrent.futures import ThreadPoolExecutor
guard = BudgetGuard(limit_usd=1.00)
llm = GuardedOpenAI(openai.OpenAI(), guard, Price(2.50, 10.00))
def agent(i):
try:
llm.chat(model="gpt-4o",
messages=[{"role": "user", "content": f"task {i}"}],
max_completion_tokens=4000)
return "ok"
except BudgetExceeded:
return "blocked"
with ThreadPoolExecutor(max_workers=20) as pool:
results = list(pool.map(agent, range(20)))
print(results.count("ok"), results.count("blocked"), f"${guard.spent:.4f}")
assert guard.spent <= guard.limit
Each call can cost at most about $0.04 here, so the first batch reserves up to the cap, the rest are blocked right away, and the assertion holds every time. Run the same test against the naive guard and the assertion fails depending on timing, which also makes it a nasty bug to reproduce in CI.
Where this stops being enough
threading.Lock only covers a single process. Once you have several gunicorn workers, a Celery fleet, or a few containers, the reservation has to move to shared storage: a Redis Lua script that does the compare-and-increment atomically, or a Postgres UPDATE budgets SET reserved = reserved + $1 WHERE id = $2 AND spent + reserved + $1 <= cap_usd RETURNING ... that affects zero rows when you're over. The logic is the same; only the lock changes.
You'll also want holds to expire. A worker that crashes between reserve and settle leaves budget locked up permanently unless something sweeps stale reservations.
I packaged this pattern as baar-core, an open-source Python library that enforces a hard cap on each LLM call and raises a 402 before the provider is contacted, using atomic reservation so concurrent calls can't jointly overshoot. pip install baar-core if you'd rather not maintain the edge cases yourself. For teams that need per-user budgets across a SaaS product, noburn.dev builds on it, enforcing limits pre-flight so a user who has exceeded their budget has their API calls blocked before they fire.
Whichever you use, the important thing is that the check runs before the request goes out. Anything after that is just reporting.
What's your worst-case max_tokens in production right now, and could you actually pay for twenty of those running at once?
Originally published at https://www.robatdasorvi.com/chapters/building/why-i-built-hard-spending-caps-for-llm-apis
Top comments (0)