A 429 at 02:00: Postmortem for a Quota Outage

Your phone buzzes at 02:07 with a failed staging job, which on its own is survivable, and then four minutes later the production summarizer starts emitting truncated output for real users. You check the model provider's status page and it is a wall of green, so you conclude the model is fine and start hunting through your own prompts instead. That instinct costs you twenty minutes, because the outage was never a model problem — it was a credential and retry problem that the model was merely the first to notice.

A boundary before we go further: this is a teaching postmortem, reconstructed from a well-documented failure class rather than a report on one named company's incident. The timeline is a plausible sequence, the code below is runnable against a fake provider, and the only numbers you should trust are the ones you print on your own machine. That distinction matters, because postmortems copied verbatim from someone else's outage teach you nothing about your own blast radius.

Timeline

Time	Observation
02:00	Nightly staging suite begins; it imports the same client library and the same API credential as production.
02:03	Staging's summarization fixtures exhaust the shared quota; the provider starts returning 429.
02:07	Production requests begin returning 429 as well, and the client treats them as generic errors.
02:11	Retry amplification peaks: every in-flight worker retries on the same fixed one-second boundary.
02:19	On-call inspects prompts and model selection, because error dashboards only count 5xx responses.
02:34	A per-environment key rotation in staging stops the bleeding; production recovers without a deploy.
03:10	First honest summary: the feature degraded because our retry policy converted a soft limit into a hard outage.

Contributing factors, in order of leverage

The first factor is shared credentials across environments, which turns a test suite into a production dependency without anyone deciding that on purpose. Nobody wrote a ticket saying “let staging rate-limit production,” but that is precisely what the shared key encoded, and it stayed invisible because both environments succeeded for months. When you cannot attribute a request to an environment, you also cannot revoke or budget it independently.

The second factor is retry without jitter, which is the classic way to convert a temporary throttle into a sustained stampede. Fixed delays look harmless in a single-threaded test and become a synchronized heartbeat in production, where dozens of workers wake at the same millisecond and fire again. Exponential backoff helps only when you also randomize the interval and cap the number of attempts; otherwise you have made each individual client polite while making the fleet as rude as before.

The third factor is alerting that keyed on 5xx only, which meant a 429 looked like success to the dashboard and like failure to the user. Quota pressure is a leading indicator, so if your error-rate panel ignores status 429 you will always learn about exhaustion from the customers instead of the graph. The uncomfortable lesson is that our observability was tuned for crashes, not for throttling.

The fix you can rehearse

The durable fix has four parts, and only the first is infrastructure: environment-scoped keys, a client-side token budget with a hard ceiling, jittered retries with a small attempt cap, and a circuit breaker that fails fast once throttling is sustained. The script below models the difference between a naive client and a jittered client behind a breaker, using a deterministic fake provider so the result is reproducible on any laptop.

"""quota_sim.py — rehearse a retry storm against a capped fake provider.
Run: python quota_sim.py
"""
from dataclasses import dataclass, field
import random


@dataclass
class CappedProvider:
    """Accepts a fixed number of call-units, then returns 429 forever."""
    capacity: int = 100
    cost_per_call: int = 12
    seen: int = field(default=0)
    accepted: int = field(default=0)

    def complete(self) -> int:
        self.seen += 1
        self.capacity -= self.cost_per_call
        if self.capacity < 0:
            return 429
        self.accepted += 1
        return 200


class Breaker:
    """Opens after `threshold` consecutive throttles, stays open for the run."""

    def __init__(self, threshold: int = 4) -> None:
        self.threshold = threshold
        self.failures = 0
        self.open = False

    def record(self, status: int) -> None:
        if status == 429:
            self.failures += 1
            if self.failures >= self.threshold:
                self.open = True
        else:
            self.failures = 0


def naive_client(provider: CappedProvider, rng: random.Random) -> int:
    """Fixed one-second retry, no jitter: every worker resumes together."""
    for _ in range(6):
        status = provider.complete()
        if status == 200:
            return status
        # time.sleep(1.0) — identical across all workers
    return 429


def jittered_client(provider: CappedProvider, rng: random.Random) -> int:
    """Full jitter, three attempts, then a breaker that fails fast."""
    breaker = Breaker()
    for attempt in range(3):
        if breaker.open:
            return 429
        status = provider.complete()
        breaker.record(status)
        if status == 200:
            return status
        base = 0.2 * (2 ** attempt)
        delay = min(2.0, base) * rng.uniform(0.5, 1.5)  # time.sleep(delay)
        _ = delay
    return 429


if __name__ == "__main__":
    for name, fn in (("naive", naive_client), ("jitter+breaker", jittered_client)):
        provider = CappedProvider()
        rng = random.Random(11)
        codes = [fn(provider, rng) for _ in range(40)]
        amplification = provider.seen / max(provider.accepted, 1)
        print(
            f"{name:>15}: requests={provider.seen:>4} "
            f"ok={provider.accepted:>3} failed={codes.count(429):>3} "
            f"amplification={amplification:.2f}"
        )

The invariant worth asserting in CI is directional rather than numeric: the jittered client behind a breaker should issue strictly fewer requests per successful call than the naive client, and it should stop issuing requests entirely once the breaker opens. Your absolute amplification depends on capacity, worker count, and cost per call, so treat the printed table as a knob you tune rather than a benchmark to quote. Add a test that fails when amplification for the naive strategy drops below the jittered one, and you have encoded the lesson instead of just writing it down.

Where a free tier belongs in this workflow

Here is the part that is genuinely useful: you want to rehearse this failure class regularly, and rehearsal that spends real money is rehearsal that quietly stops happening. MonkeyCode offers free model access and a free server option that the project documents on its own pages, which makes it a reasonable place to run disaster drills like the one above without treating production budget as the test fixture. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I have not measured its quotas, hardware, or long-term availability, and free-tier terms change, so read the current allocation on the project site rather than trusting any number quoted in a blog post, including this one.

Workload	Sensible home	Reasoning
Retry-storm drills, breaker tests	Free tier	Failure is the point; nothing of value is lost when the drill trips a limit.
Nightly staging suites	Free tier, separate credentials	Isolates test load from anything a customer can feel.
Prompt-regression evals on synthetic data	Free tier	Cheap, repeatable, and easy to throw away when the schema changes.
User-facing generation with an SLO	Paid, environment-scoped keys	Throttling here is an outage, not a signal.
Anything bound by a data-processing agreement	Whatever your legal review approved	Free hosting does not change your obligations.

Limitations and who should not do this

A circuit breaker is a blunt instrument that can mask a genuine provider-side regression, because once it opens you stop collecting the evidence you would need to tell throttling apart from a broken deploy. If your team has nobody who owns the shared client library, adding a token budget and breaker to it will produce three divergent copies within a quarter, and the drift will be worse than the original bug. Per-environment keys also add rotation chores, so a two-person team should weigh whether a shared key plus a hard client-side ceiling is enough for now.

You should not adopt this pattern if you need guaranteed throughput, if your latency budget cannot absorb a fail-fast response, or if your prompts carry data your agreements forbid sending to a third-party endpoint of any price. Free model access is excellent for rehearsal and poor for contractual guarantees, and pretending otherwise is how you end up writing a second postmortem about the same night. For everyone else, the sequence is unglamorous: scope the credentials, budget the tokens, jitter the retries, break the circuit, and alert on 429 before your users do.

The 02:00 page was never about which model you picked, and the fix that survived review was fourteen lines in a client library that nobody had looked at in six months. If you want to rehearse that night without spending anything, the free tier is a reasonable place to start, and you can keep the simulation in your repository as the artifact that proves you learned it.