The Retry Loop That Turned a 429 Into an Outage

The outage that woke me at 2 AM was not caused by a bad model or an overloaded server. It was caused by a retry loop that treated a token-limit rejection as a transient network error. The same pipeline had passed every eval we ran, which made the failure harder to diagnose rather than easier. We trusted the green checks and stopped questioning them. The exact numbers here are disguised, but the pattern is one I have seen repeat across several projects.

Production started failing around 1:47 AM with a pattern that looked random. Roughly one in six requests to our summarization job returned HTTP 429, and a smaller number timed out after ninety seconds. The job was small, a background worker that condensed support threads into a daily digest, and it had run for two weeks without a single recorded error. By the time I opened the logs, the retry counter had already pushed the same failed payload through the queue eleven times. The failure was not an event; it was a loop.

Our first hypothesis was the usual suspect: the free tier had hit its rate limit. We added a sleep between requests and waited for the pattern to disappear, but the failures continued at the same frequency. That ruled out a simple per-minute quota, so the second hypothesis blamed the network, and we switched providers only to reproduce the identical behavior. The turning point came when we logged the exact request body alongside the status code. The response body contained a detail we had never parsed: the server rejected the request based on the token count it computed, not the count we estimated.

# Inspect the rejection instead of guessing from the status code
curl -i "$ENDPOINT" \
  -H "Authorization: Bearer $TOKEN" \
  -d @payload.json | head -40

Here is the gap: our client-side tokenizer estimated 8,400 tokens for the digest prompt. The server counted 10,300 because it included the full conversation history we appended in a separate step. The request crossed the allowance threshold, the server returned 429 with a message that looked like a rate limit, and our retry logic dutifully re-sent the same oversized payload. That single decision turned one contract violation into eleven identical violations. The synchronized backoff then made every worker hammer the endpoint at the same moment.

# The original retry: same payload, synchronized backoff, no jitter
for attempt in range(10):
    response = call_model(payload)
    if response.status_code == 429:
        time.sleep(2 ** attempt)  # every worker sleeps the same schedule
        continue
    break

The fix had three parts, and none of them involved changing the model. First, we parsed the response body on every non-200 status and logged the server's stated token count. A contract rejection became a number we could compare against our own estimate. Second, we added a client-side hard cap that counted the full conversation history with the same rules the server used. We rejected the job before it ever reached the network. Third, we replaced the synchronized backoff with a bounded retry that added random jitter and capped the total attempts at three.

# The fixed version: parse the reason, cap the payload, jitter the retry
def call_with_budget(payload, max_tokens):
    if estimate_full_payload(payload) > max_tokens:
        raise BudgetExceeded(payload)  # fail before the network
    for attempt in range(3):
        response = call_model(payload)
        if response.status_code == 429:
            reason = response.json().get("error", {}).get("message", "")
            log("429", reason, estimate_full_payload(payload))
            time.sleep((2 ** attempt) + random.uniform(0, 1.5))
            continue
        return response
    raise RetryBudgetExhausted(payload)

To verify the fix without burning paid credits, we rebuilt the same job on a free server and pointed it at MonkeyCode's open-source project, whose free model access currently includes a ten-million-token allowance and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The goal of that exercise was not to benchmark the provider but to reproduce the exact failure mode in a controlled environment. The free allowance made it practical to run the reproduction dozens of times in a single afternoon. Free quotas change, so check the current terms before you rely on the numbers, and treat any free tier as a sandbox rather than a guarantee.

The reusable lesson is to treat every error code as a hypothesis, not a conclusion. A 429 can mean a rate limit, a quota, a payload violation, or a server-side rejection, and the response body usually tells you which one if you bother to read it. Logging that body is the cheapest instrumentation you will ever add. The second lesson is that retry logic is part of your system's behavior, so a failure that repeats eleven times is a design flaw, not bad luck. Jitter exists for a reason, and bounded retries exist for a better one.

This approach is not for everyone, and you should skip it if your workload sends a handful of requests per day and you can inspect each failure by hand. A client-side token cap will not save you from a model that suddenly changes its output length, so you still need output validation for real-time user-facing features. And if you are evaluating a model for production, a free tier is a useful sandbox, not a substitute for a contract test against the provider you will actually pay for.

The next time a background job fails at 2 AM, read the response body before you touch the retry loop, because the model was rarely the problem. The fix was a few lines of logging, a hard cap, and a jittered retry, and you can test all of it on a free server without spending a cent. That is the kind of debugging you can do anywhere, and a free tier just happens to make it cheaper to practice.