My first real LLM bill surprise wasn't dramatic. No infinite loop, no viral spike. A retry policy I'd written months earlier met a flaky endpoint, and a background job quietly re-sent the same long prompt all night. The bill was just... 40x normal. Nothing "failed", so nothing alerted.
I've spent the last few weeks building a hard spending cap for LLM calls, and most of what I learned wasn't about caps at all. It was about how many small boring ways money leaks when the meter only exists on the provider's side. Here's the list I wish someone had handed me.
1. Billing dashboards are rear-view mirrors
Every provider dashboard answers "how much did you spend?" hours later, after aggregation. Nobody answers "how much are you spending right now, from which feature?" By the time the dashboard shows the leak, the leak already happened. If you take one thing from this post: put the meter inside your process, at the call site, where it can act before the request instead of reporting after it.
2. Total spend is a useless number
"$120 today" tells you nothing actionable. The question that matters is which feature is leaking. The fix is one tag per call (feature: "chat", feature: "summarizer") and a per-feature breakdown. The first time I ran a breakdown on a real app, a background enrichment job I'd forgotten about turned out to be 60% of spend. For me, the per-feature breakdown was the entire diagnosis.
3. Every provider reports usage differently, and they all lie a little
OpenAI gives you prompt_tokens and completion_tokens. Anthropic gives input_tokens/output_tokens. Gemini nests everything in usageMetadata. Bedrock and Cohere have their own shapes. Then it gets worse: cached input tokens bill at a different rate, and reasoning tokens are billed as output even though you never see them. Two providers even disagree on whether reasoning tokens are counted inside the output number at all. xAI and Gemini report them outside it, so if you naively bill the output count you're undercounting exactly when reasoning models get expensive. Budget a week of reading API docs that contradict the actual responses.
4. Streaming hides the receipt until the very end
With stream: true, usage arrives (if at all) in the final chunk. OpenAI won't even send it unless you pass stream_options: { include_usage: true }. Anthropic splits it across message_start and message_delta events you have to assemble, and the delta is cumulative, so adding instead of replacing double-counts. Gemini sends a running usageMetadata where only the last value is real. Any cost tracking that ignores streaming is blind to the calls most modern apps actually make.
5. A cap that checks after the call is a receipt, not a guard
The naive cap goes: call, add to a counter, compare. It always overshoots by one call. Concurrency makes it much worse. Ten parallel requests all read the same counter, all pass the check, all land. I only really fixed this with an atomic reserve-then-settle: estimate the cost, reserve it against the cap before the call (one atomic operation, Lua script if the counter lives in Redis), settle the difference after. If you run multiple workers the counter has to live in shared storage anyway, otherwise each worker politely enforces its own private budget while the fleet blows through ten of them.
6. Retry storms are the leak nobody instruments
The single most expensive bug class I found wasn't prompts. It was retries: exponential backoff on a 429 while the request is actually succeeding on the provider's side, queue re-drives, at-least-once delivery re-running completed jobs. Track consecutive failures per feature and alert when the streak gets weird. A hard cap turns this failure mode from "unbounded bill" into "capped error you notice in the morning", and that trade is almost always correct for background workloads.
7. Caps and traces are two sides of the same thing
A reader of my first post put it better than I had: cost caps and execution traces are both answers to "what is my agent doing when I'm not looking?" The cap is the brake. A per-call spend event (feature, model, usd, running total) piped into the logs you already have is the gauge. You want both, because a brake without a gauge just means you get stopped without knowing why.
I ended up packaging all of this into a small MIT-licensed library called budget-guard: zero-dependency hard caps that hold under concurrent workers, per-feature attribution, streaming usage, and adapters for Vercel AI SDK, LangChain.js, LlamaIndex.TS and Mastra. But honestly, even if you never install anything: tag your calls, meter before the request, and go look at what your retries did last night.
Top comments (11)
Tagging every call by feature is the detail most people skip, and your enrichment job being 60% of spend is exactly why a single total number hides the actual leak. Putting the meter at the call site rather than trusting the provider dashboard also gives you a place to enforce a hard stop before the request goes out, not just an after-the-fact report. The differing token shapes across providers is a trap, so normalizing them into one internal unit early saves a lot of reconciliation later.
Normalizing early is the right call, and it's also where the bodies are buried. I ended up making USD the internal unit rather than tokens, because tokens stopped being comparable once cached input and reasoning tokens got their own rates. One "token" can be four different prices inside a single response. The raw counts are still worth logging for the day a number looks wrong, but caps, alerts and the per-feature breakdown all run on dollars.
The word silently is the key. Most teams can handle expensive runs if they see them early and can tie them to a task. The dangerous costs are the ones hidden inside retries, long context, broad tool schemas, and loops that look productive until the invoice arrives.
Yeah, "looks productive" is the scary part. The job that blew my bill up was green in every dashboard we had. The only thing that knew was the billing page.
That is the exact dashboard trap. Green system metrics can hide a bad business loop if nobody is watching cost per useful output. I like treating billing as one more production signal, not a finance surprise at the end of the week.
That is the worst version of the problem: a green system that is quietly expensive. I like treating cost as an observable behavior, not just a billing artifact discovered after the run.
The meter-belongs-inside-the-process point is exactly the lesson most teams learn the painful way. The retry-on-flaky-endpoint case you describe is particularly insidious because the job reports success from its own perspective - no errors thrown, no alerts fired. Two additions that made a big difference in practice: circuit breakers that check accumulated cost against a per-session budget before allowing a retry, and a dead-letter queue for prompts that hit the budget ceiling so they can be inspected rather than silently dropped. The feature-level tagging suggestion is underrated and probably the highest-ROI change in the list.
Quick follow-up: the onRejected hook just shipped in v0.7.0. Blocked calls now hand you the original request, so they can go to a dead-letter queue or a replay pile instead of vanishing. There's also an onThreshold warning at 80% of the cap now, which is half of your circuit-breaker idea baked in. Thanks for the push.
The dead-letter idea is the best suggestion this post has gotten. Right now a rejected call just throws, so the prompt evaporates unless the caller kept a copy, and nobody keeps a copy. I opened an issue for an onRejected hook so rejected calls land somewhere inspectable: github.com/kimbeomgyu/budget-guard...
Your circuit breaker is roughly where I ended up too, just folded into the cap itself: reserve the estimated cost atomically before the call, settle the real number after. A retry loop then dies on the reservation instead of ten requests later. And yes on tagging. One line per call, and it found me a forgotten background job eating 60% of spend.
retries are the one people never see coming.. a timeout plus an automatic retry on a long context call quietly doubles the bill and shows up as a shape in the invoice nobody can explain for weeks. which of the seven actually moved the number.. in my experience one or two of these account for most of the saving and the rest is noise?
Honest answer: two did most of it. Per-feature tagging found the actual leak, a background enrichment job at 60% of spend that no dashboard had a name for. The hard cap turned the worst case from an invoice surprise into a blocked call. The other five mostly exist so those two keep working under real conditions.
The retry shape you describe is the one that made me add storm detection: consecutive provider failures per feature and model fire a callback before the loop re-burns the budget. A timeout plus auto-retry on a long context call is exactly what it exists for. And since the cap started reserving the estimated cost before each call, a retry loop dies on the reservation instead of ten requests later.