A budget that reports at the end of the period is accounting.
A permission constrains the future, before the call.
The concurrency failure is the whole story.
5 workers check whether $4000 is under a $5000 cap, each sees yes, each spends $1200. Every individual check was correct.
Reserve in-flight calls instead:
def authorize(cost_cents, period):
spent = counters.spent(period)
held = reservations.outstanding(period)
if spent + held + cost_cents > CAP:
return Decision("block")
reservations.create(cost_cents, period, ttl=300)
return Decision("allow")
Two settlement skews to design for up front: actual cost differing from estimate, and the settlement write failing outright.
Both are ordinary.
Full writeup with the field-classification problem: here
Top comments (1)
The reservation framing is the right one. One thing worth checking in the snippet though: reading spent + held and then calling reservations.create are two separate steps, so five workers can still all pass the check before any of them writes a hold. It has to be one atomic step, like a single SQL transaction with a row lock on the period counter, or a Lua script if the counters live in Redis.
On settlement skew, I'd be careful with what the TTL does on expiry. If the settlement write failed but the call actually ran, silently releasing the hold undercounts. Moving it to an 'unsettled' state that still counts against the cap until you reconcile with provider usage seems safer.
How are you estimating cost_cents up front for LLM calls? max_tokens times the output price, then refunding the difference at settlement?