My first real LLM bill surprise wasn't dramatic. No infinite loop, no viral spike. A retry policy I'd written months earlier met a flaky endpoint, ...
For further actions, you may consider blocking this person and/or reporting abuse
Tagging every call by feature is the detail most people skip, and your enrichment job being 60% of spend is exactly why a single total number hides the actual leak. Putting the meter at the call site rather than trusting the provider dashboard also gives you a place to enforce a hard stop before the request goes out, not just an after-the-fact report. The differing token shapes across providers is a trap, so normalizing them into one internal unit early saves a lot of reconciliation later.
Normalizing early is the right call, and it's also where the bodies are buried. I ended up making USD the internal unit rather than tokens, because tokens stopped being comparable once cached input and reasoning tokens got their own rates. One "token" can be four different prices inside a single response. The raw counts are still worth logging for the day a number looks wrong, but caps, alerts and the per-feature breakdown all run on dollars.
The word silently is the key. Most teams can handle expensive runs if they see them early and can tie them to a task. The dangerous costs are the ones hidden inside retries, long context, broad tool schemas, and loops that look productive until the invoice arrives.
Yeah, "looks productive" is the scary part. The job that blew my bill up was green in every dashboard we had. The only thing that knew was the billing page.
That is the exact dashboard trap. Green system metrics can hide a bad business loop if nobody is watching cost per useful output. I like treating billing as one more production signal, not a finance surprise at the end of the week.
That is the worst version of the problem: a green system that is quietly expensive. I like treating cost as an observable behavior, not just a billing artifact discovered after the run.
The meter-belongs-inside-the-process point is exactly the lesson most teams learn the painful way. The retry-on-flaky-endpoint case you describe is particularly insidious because the job reports success from its own perspective - no errors thrown, no alerts fired. Two additions that made a big difference in practice: circuit breakers that check accumulated cost against a per-session budget before allowing a retry, and a dead-letter queue for prompts that hit the budget ceiling so they can be inspected rather than silently dropped. The feature-level tagging suggestion is underrated and probably the highest-ROI change in the list.
Quick follow-up: the onRejected hook just shipped in v0.7.0. Blocked calls now hand you the original request, so they can go to a dead-letter queue or a replay pile instead of vanishing. There's also an onThreshold warning at 80% of the cap now, which is half of your circuit-breaker idea baked in. Thanks for the push.
The dead-letter idea is the best suggestion this post has gotten. Right now a rejected call just throws, so the prompt evaporates unless the caller kept a copy, and nobody keeps a copy. I opened an issue for an onRejected hook so rejected calls land somewhere inspectable: github.com/kimbeomgyu/budget-guard...
Your circuit breaker is roughly where I ended up too, just folded into the cap itself: reserve the estimated cost atomically before the call, settle the real number after. A retry loop then dies on the reservation instead of ten requests later. And yes on tagging. One line per call, and it found me a forgotten background job eating 60% of spend.
retries are the one people never see coming.. a timeout plus an automatic retry on a long context call quietly doubles the bill and shows up as a shape in the invoice nobody can explain for weeks. which of the seven actually moved the number.. in my experience one or two of these account for most of the saving and the rest is noise?
Honest answer: two did most of it. Per-feature tagging found the actual leak, a background enrichment job at 60% of spend that no dashboard had a name for. The hard cap turned the worst case from an invoice surprise into a blocked call. The other five mostly exist so those two keep working under real conditions.
The retry shape you describe is the one that made me add storm detection: consecutive provider failures per feature and model fire a callback before the loop re-burns the budget. A timeout plus auto-retry on a long context call is exactly what it exists for. And since the cap started reserving the estimated cost before each call, a retry loop dies on the reservation instead of ten requests later.