Postmortem: The Night the Free Tier Burned the Whole Budget

It is 03:12 and your phone lights up with a spend alert for a service that was designed to cost nothing at all. The token counter on the dashboard is climbing in a perfectly straight line, the error rate is flat, and every individual request in the log looks completely normal. This is the quietest kind of incident, because nothing is down, no user is paged, and the only signal is money leaving faster than work arrives.

What follows is a postmortem you can reproduce rather than a retelling of someone else's outage. The timeline and the two fixes below are modeled on a small fault-injection harness you can run locally in about ten minutes. Where your reproduction diverges from this account, that divergence is the most useful thing you will learn today, so keep a terminal open while you read.

Timeline of a flat-looking failure

At 03:12 the upstream provider starts dropping roughly one connection in three, and each drop happens after the completion has already been generated and billed. Your worker sees a timeout, treats it as "the request never happened", and retries the identical prompt immediately with no backoff and no idempotency key. By 03:19 the retry storm is self-sustaining: each real request now costs about 1.5 completions, and the extra load nudges the upstream into dropping more connections, which produces yet more retries.

At 03:41 the queue depth has doubled while the success rate has halved, so the autoscaler adds workers, which is exactly the wrong response. At 04:02 the spend counter crosses the budget you believed was enforced by the provider's free allowance, and the alert finally fires. Notice that the latency graph looks fine throughout, because the retried requests that do succeed are fast and the ones that time out are invisible.

Three contributing factors, none of them exotic

The first factor is retrying a non-idempotent call. A completion request is not a GET, and the cheap mental model of "try again if it fails" hides the fact that the expensive work already happened server-side before your socket died.

The second factor is observation without reservation. You measured tokens after the response came back, so your budget check could only ever describe the past and never refuse the future. A spend counter that is read-only is a dashboard, not a control.

The third factor is unbounded concurrency per queue item. One worker with four retries and a queue that itself retries failed items multiplies into sixteen attempts for a single user action, and nothing stood between the queue and the provider. These three combine into the classic shape of a free-tier incident: the tier is generous, your client is greedy, and the gap stays invisible until the counter runs.

The harness that reproduces it

Save two files: a stub gateway that bills work and then fails a share of connections, and a client that retries without keys. The gateway is deliberately dumb, because you want the failure to be deterministic rather than mysterious.

# stub_gateway.py - bills first, then drops the socket, like a real provider
import random
from fastapi import FastAPI

app = FastAPI()
SPENT = {"tokens": 0}

@app.post("/v1/complete")
def complete(body: dict):
    cost = len(body.get("prompt", "")) // 4 + 40
    SPENT["tokens"] += cost              # work done, invoice written
    if random.random() < 0.30:
        raise TimeoutError("socket dropped after generation")
    return {"usage": {"total_tokens": cost}}

@app.get("/v1/spend")
def spend():
    return SPENT

The client below is the bug in twelve lines. It retries on timeout, it never sends an idempotency key, and it has no idea how much it has already spent.

# repro.py - the buggy client; run it and watch /v1/spend
import asyncio, sys, httpx

BASE = "http://127.0.0.1:8080"

async def call(client, prompt, attempt=1):
    try:
        r = await client.post(f"{BASE}/v1/complete", json={"prompt": prompt}, timeout=6.0)
        r.raise_for_status()
        return r.json()["usage"]["total_tokens"]
    except (httpx.TimeoutException, httpx.HTTPStatusError):
        if attempt >= 4:
            raise
        return await call(client, prompt, attempt + 1)   # <- the whole incident

async def main(n):
    async with httpx.AsyncClient() as client:
        results = await asyncio.gather(
            *[call(client, "classify this ticket " * 40) for _ in range(n)],
            return_exceptions=True,
        )
    ok = sum(1 for r in results if isinstance(r, int))
    print(f"delivered={ok} failed={n - ok}")

asyncio.run(main(int(sys.argv[1]) if len(sys.argv) > 1 else 200))

Run uvicorn stub_gateway:app --port 8080 in one terminal and python repro.py 200 in another, then read the invoice with curl -s 127.0.0.1:8080/v1/spend. Expect spend to exceed delivered work, and expect that ratio to worsen as you raise the concurrency argument, which is the signature of the incident above.

The durable fix: reserve, key, and degrade

A budget you cannot refuse is not a budget, so the fix moves the decision before the call. Reserve an estimated cost, settle the actual usage afterwards, and let admission control answer "no" when the reserve would overshoot the cap. This is the same idea as a database connection pool, just applied to tokens instead of sockets.

class BudgetExhausted(Exception):
    pass

class Ledger:
    def __init__(self, cap_tokens: int):
        self.cap, self.spent = cap_tokens, 0

    def reserve(self, estimate: int) -> int:
        if self.spent + estimate > self.cap:
            raise BudgetExhausted(f"{self.spent}+{estimate} > {self.cap}")
        self.spent += estimate
        return estimate

    def settle(self, estimate: int, actual: int) -> None:
        self.spent += actual - estimate

Wire it in with an idempotency key derived from stable inputs, so a retry of the same logical request can be deduplicated or served from cache instead of regenerated. Keep exactly one retry with jitter, and on exhaustion prefer a cheaper path over another attempt.

key = hashlib.sha256(f"{user_id}:{task_id}:{prompt_version}".encode()).hexdigest()
estimate = ledger.reserve(rough_tokens(prompt))
try:
    usage, text = await complete(prompt, idem_key=key, retries=1)
    ledger.settle(estimate, usage)
except BudgetExhausted:
    text = cheap_fallback(prompt)      # tier 3: no retry, no escalation

The third piece is a tier table you can defend in a review, because "use the cheap model when busy" is not a policy anyone can test. Write down which class of work each tier may serve, and make escalation an explicit decision rather than an automatic reflex.

Tier	Serves	Retries	On budget pressure
1	user-visible, latency-bound	1, jittered	queue it; never silently downgrade
2	batch summarisation, tagging	1	pause until the next window
3	bulk classification, dedupe	0	serve the degraded result, log it

Tier three is where free model access genuinely earns its place, because bulk classification is exactly the workload whose cost you cannot predict and whose latency nobody feels. MonkeyCode is one option for that tier: it offers free model access and a free server option, and the operator states a free allowance of ten million tokens. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server matters less for benchmarking and more for keeping the ledger awake, because a budget check that dies with your laptop is the same read-only dashboard you had before. Put the reserve-and-settle wrapper plus a tiny spend endpoint on that box, point every worker at it, and the counter becomes an admission gate instead of a postmortem exhibit.

What this does not fix

Free allowances change, so pin the current terms in a config comment and re-read them before you design a budget around that ten-million figure; treat any published number as a today-number rather than a contract. This design also assumes your provider or your cache actually honors idempotency keys, and if it does not, you need a content-addressed cache in front of the call or you are back to retrying blind. Finally, free model access in the request path is reasonable for tier-three bulk work and unreasonable for anything with an SLA, a hard latency target, regulated data, an audit trail, or long-running stateful jobs.

If you want to test the third tier cheaply, run the harness against your own stub first, then swap in the free model access and the free server option and keep the same thirty percent drop rate. If the ledger stays flat while delivery stays level, you have just prevented an incident instead of writing about one, which is the only postmortem worth publishing.