What Happens When Your AI API Returns a 429? A Retry Strategy That Actually Works

Last month, I burned through $200 in AI API credits in a single afternoon. Not from heavy usage — from retries gone wrong.

I had a simple while True: try/except loop calling five different LLM providers. When one returned a 429 (rate limit), my script panicked and retried immediately. When another returned a 503, it retried immediately too. By the time I noticed, my OpenAI bill had a spike that looked like a mountain range.

Here's what I built to fix it — and what I learned along the way.

---

The Problem: Default Retries Are Dumb

Most HTTP libraries come with some kind of retry mechanism. Requests has Retry. httpx has limits. But here's the thing — they all retry the same way for every error.

A 429 (rate limit) and a 503 (temporary overload) look the same to a generic retry loop. They shouldn't.

Status	Meaning	Smart Response
429	You're sending too fast	Back off — the server told you to wait
503	Server is overloaded	Wait and retry — it might clear up
500	Internal server error	Maybe retry — could be transient
401	Invalid API key	Stop immediately — this won't fix itself
400	Bad request	Stop immediately — you sent bad data

A dumb retry loop treats them all the same. A smart one reads the status code and acts accordingly.

---

Step 1: Classify Before You Retry

Before you even think about backing off, categorize the error:

from enum import Enum

class ErrorStrategy(Enum):
    RETRY = "retry"           # transient, safe to retry
    RETRY_WITH_BACKOFF = "backoff"  # rate-limited, must wait
    ABORT = "abort"           # permanent, don't retry

def classify_status(status: int) -> ErrorStrategy:
    if status == 429:
        return ErrorStrategy.RETRY_WITH_BACKOFF
    if status in (503, 502, 504):
        return ErrorStrategy.RETRY
    if status >= 500:
        return ErrorStrategy.RETRY  # servers recover
    if status in (401, 403, 400):
        return ErrorStrategy.ABORT
    return ErrorStrategy.ABORT  # unknown = don't guess

This alone saved me dozens of wasted calls. No more retrying with an expired API key.

---

Step 2: Exponential Backoff With Jitter

The standard advice is "use exponential backoff." That means waiting 1s, then 2s, then 4s, then 8s. But there's a problem.

Imagine 100 clients all hit a rate limit at the same time. They all wait 1 second and retry — creating a second spike exactly 1 second later. Then they all wait 2 seconds and spike again. This is called the thundering herd problem.

The fix is jitter — adding randomness to the wait time:

import random
import time

def backoff_with_jitter(attempt: int, base: float = 1.0, max_wait: float = 60.0) -> float:
    """Exponential backoff with full jitter.

    Instead of wait = base * 2^attempt, we pick a random value
    between 0 and base * 2^attempt. This spreads retries across
    the interval and eliminates synchronized spikes.
    """
    wait = min(base * (2 ** attempt), max_wait)
    return random.uniform(0, wait)

I learned this the hard way after watching three worker processes synchronize their retries perfectly — all firing at the exact same millisecond, every cycle. Jitter broke the pattern instantly.

---

Step 3: Read the Retry-After Header

Most well-behaved APIs send a Retry-After header with 429 responses. Use it:

def get_retry_delay(response, attempt: int) -> float:
    """Respect the server's Retry-After header when available."""
    retry_after = response.headers.get("Retry-After")
    if retry_after is not None:
        try:
            return float(retry_after)  # seconds
        except ValueError:
            pass  # might be a date, skip

    # Fall back to exponential backoff with jitter
    return backoff_with_jitter(attempt)

OpenAI, Anthropic, and Google all return Retry-After. Respecting it means you're a good API citizen — and you get unblocked faster.

---

Step 4: Add a Circuit Breaker

After enough failures, stop trying. A circuit breaker prevents cascading failures:

import time
from dataclasses import dataclass, field

@dataclass
class CircuitBreaker:
    failure_threshold: int = 5
    recovery_timeout: float = 30.0  # seconds
    failures: int = 0
    last_failure_time: float = 0.0
    is_open: bool = False

    def record_failure(self):
        self.failures += 1
        self.last_failure_time = time.time()
        if self.failures >= self.failure_threshold:
            self.is_open = True

    def allow_request(self) -> bool:
        if not self.is_open:
            return True
        if time.time() - self.last_failure_time > self.recovery_timeout:
            # Try again — half-open state
            self.is_open = False
            self.failures = 0
            return True
        return False

When the circuit is open, my code doesn't even attempt the API call. It returns a cached fallback or a graceful error. No more burning credits on a provider that's clearly down.

---

Putting It All Together

Here's the complete retry wrapper I use now:

import httpx
import random
import time
from typing import Optional

def call_with_smart_retry(
    url: str,
    headers: dict,
    payload: dict,
    max_attempts: int = 5,
    circuit_breaker: Optional[CircuitBreaker] = None,
) -> dict:
    for attempt in range(max_attempts):
        if circuit_breaker and not circuit_breaker.allow_request():
            raise Exception(f"Circuit breaker open for {url}")

        try:
            response = httpx.post(url, json=payload, headers=headers, timeout=30)
        except httpx.TimeoutException:
            wait = backoff_with_jitter(attempt, base=2.0)
            time.sleep(wait)
            continue

        strategy = classify_status(response.status_code)

        if strategy == ErrorStrategy.ABORT:
            response.raise_for_status()

        if response.is_success:
            if circuit_breaker:
                circuit_breaker.failures = 0  # reset on success
            return response.json()

        if strategy == ErrorStrategy.RETRY_WITH_BACKOFF:
            delay = get_retry_delay(response, attempt)
            time.sleep(delay)
        elif strategy == ErrorStrategy.RETRY:
            wait = backoff_with_jitter(attempt)
            time.sleep(wait)

        if circuit_breaker:
            circuit_breaker.record_failure()

    raise Exception(f"Failed after {max_attempts} attempts")

---

What I Learned

1. Classify errors before retrying. Not all failures are temporary. A 401 doesn't become a 200 if you wait long enough.

2. Use jitter, always. Without it, your retries will synchronize. I watched it happen — three workers hitting the same API at exactly the same millisecond, every cycle.

3. Respect Retry-After headers. The server already told you how long to wait. Listen to it.

4. Circuit breakers prevent disaster. When a provider is down, stop calling it. Your retries won't fix their infrastructure.

5. Log everything. I added structured logging to every retry attempt. A month later, those logs showed me exactly which providers had the worst reliability — data I used to make routing decisions.

That $200 afternoon taught me more about API resilience than any blog post. The fix took about 100 lines of Python. Worth every cent.

---

What's your retry strategy look like? I'm curious how others handle this — drop a comment.