A 429 Is a Signal, Not a Suggestion: Build a Retry Ladder for Free Endpoints

Retries are a schedule, not a reflex. A free model endpoint returns 429 when it is overloaded. An immediate retry adds to that overload. A retry ladder spaces attempts out with exponential backoff and jitter. This tutorial builds one from zero. It adds a circuit breaker and an idempotency guard. Every stage ends with a runnable verification step. By the end you have a reusable Python module. It survives rate limits, overloads, and timeouts.

Why naive retries fail

Free endpoints fail in predictable patterns. Rate limiters return 429. Proxies return 503. Slow inference causes timeouts. Each failure type needs a different response.

A 429 without backoff becomes a thundering herd. Ten clients retry at the same instant. The endpoint stays overloaded. Jitter breaks the synchronization. A 503 without backoff hammers an already sick server. A timeout without verification may duplicate work. The pattern repeats across providers. Free endpoints are shared, so a bad retry costs other users too.

The decision table is the contract:

Status	Meaning	Action
429	Rate limited	Wait for Retry-After, then retry
502/503/504	Overloaded	Exponential backoff, then retry
Timeout	Unknown	Retry once, then validate
400/401/422	Bad request	Never retry
200 + invalid payload	Validation error	Never retry

Step 1: Classify failures before you write code

Write the classification as a function. It returns True only when a retry can help.

RETRYABLE = {429, 500, 502, 503, 504}

def should_retry(status):
    return status in RETRYABLE

Verification:

assert should_retry(429) is True
assert should_retry(503) is True
assert should_retry(400) is False
assert should_retry(422) is False
print("classification ok")

Step 2: Build the retry ladder

The ladder has three rules. Backoff grows exponentially. Jitter prevents synchronized retries. Retry-After overrides the backoff.

import random
import time

class RetryLadder:
    def __init__(self, max_attempts=4, base_delay=1.0, max_delay=30.0, jitter=0.3):
        self.max_attempts = max_attempts
        self.base_delay = base_delay
        self.max_delay = max_delay
        self.jitter = jitter

    def delay_for(self, attempt, retry_after=None):
        if retry_after is not None:
            return min(float(retry_after), self.max_delay)
        exp = min(self.base_delay * (2 ** attempt), self.max_delay)
        return exp * (1 + random.uniform(-self.jitter, self.jitter))

    def run(self, call):
        status, payload = None, None
        for attempt in range(self.max_attempts):
            try:
                status, payload = call()
            except TimeoutError:  # Python 3.10+; socket.timeout is TimeoutError
                status, payload = 503, {"error": "timeout"}
            if status not in RETRYABLE:
                return status, payload
            retry_after = None
            if isinstance(payload, dict):
                retry_after = payload.get("retry_after")
            delay = self.delay_for(attempt, retry_after)
            time.sleep(delay)
        return status, payload

The math is simple. Attempt 0 waits base_delay. Attempt 1 waits double. Attempt 2 waits quadruple. The jitter spreads the herd. Retry-After is a server directive, so it gets no jitter.

Verification:

ladder = RetryLadder(base_delay=1.0, jitter=0.0)
assert 1.0 <= ladder.delay_for(0) < 2.0
assert 2.0 <= ladder.delay_for(1) < 4.0
assert 4.0 <= ladder.delay_for(2) < 8.0
print("backoff curve ok")

This test pins the curve. If someone changes the multiplier, the assertion fails.

Step 3: Add a circuit breaker

A ladder handles a few failures. A storm needs a breaker. When five calls fail in a row, the breaker opens. New calls fail fast without touching the endpoint. After a cooldown, one probe request tests the water.

import time

class CircuitBreaker:
    def __init__(self, failure_threshold=5, cooldown=30.0):
        self.failure_threshold = failure_threshold
        self.cooldown = cooldown
        self.failures = 0
        self.state = "closed"
        self.opened_at = 0.0

    def allow(self):
        if self.state == "open":
            if time.time() - self.opened_at >= self.cooldown:
                self.state = "half_open"
                return True
            return False
        return True

    def record(self, ok):
        if ok:
            self.failures = 0
            self.state = "closed"
        else:
            self.failures += 1
            if self.failures >= self.failure_threshold:
                self.state = "open"
                self.opened_at = time.time()

Verification:

breaker = CircuitBreaker(failure_threshold=5, cooldown=0.01)
for _ in range(5):
    breaker.record(False)
assert breaker.state == "open"
assert breaker.allow() is False
time.sleep(0.02)
assert breaker.allow() is True
breaker.record(True)
assert breaker.state == "closed"
print("breaker ok")

Step 4: Make retries idempotent

LLM calls are not naturally idempotent. The same prompt can produce a different completion. A retried code-generation call may return a different patch. Store the first successful response by request hash. Return it on duplicate retries.

import hashlib
import json

class ResponseCache:
    def __init__(self):
        self.store = {}

    def key(self, request):
        raw = json.dumps(request, sort_keys=True).encode()
        return hashlib.sha256(raw).hexdigest()

    def get(self, request):
        return self.store.get(self.key(request))

    def put(self, request, response):
        self.store[self.key(request)] = response

Verification:

cache = ResponseCache()
req = {"prompt": "summarize this log", "max_tokens": 50}
cache.put(req, {"text": "first completion"})
assert cache.get(req) == {"text": "first completion"}
print("idempotency ok")

Step 5: Verify against a mock endpoint

Never test retries against a real endpoint first. Build a mock that returns a scripted failure sequence. This makes the test deterministic. The scripted sequence is the source of truth. It removes randomness from the test. Every run follows the same path.

import json
from http.server import BaseHTTPRequestHandler, HTTPServer

SCRIPT = [
    (429, {"error": "rate limited", "retry_after": 0.1}),
    (429, {"error": "rate limited", "retry_after": 0.1}),
    (503, {"error": "overloaded"}),
    (200, {"text": "ok"}),
]

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        code, payload = SCRIPT.pop(0) if SCRIPT else (200, {"text": "ok"})
        body = json.dumps(payload).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(body)

HTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

Run the mock in one terminal. Run the ladder in another.

import json
import urllib.error
import urllib.request

def call_model():
    req = urllib.request.Request(
        "http://127.0.0.1:8765/v1/complete",
        data=json.dumps({"prompt": "hello"}).encode(),
        headers={"Content-Type": "application/json"},
    )
    try:
        with urllib.request.urlopen(req, timeout=5) as resp:
            return resp.status, json.loads(resp.read())
    except urllib.error.HTTPError as e:
        return e.code, json.loads(e.read())
    except TimeoutError:
        raise

ladder = RetryLadder(max_attempts=4, base_delay=0.2, max_delay=2.0)
status, payload = ladder.run(call_model)
print(status, payload)

Expected output: 200 {'text': 'ok'}. The ladder survived two 429s and one 503. It waited, backed off, and landed on a success. Add a print inside call_model to count attempts. The pattern should be: fail, fail, fail, succeed.

Step 6: Run it next to a real free endpoint

The mock proves the logic. The real endpoint proves the assumptions. Point the same call_model at a free model endpoint. Keep the ladder on a small server so it runs even when your laptop sleeps. This is where MonkeyCode's free server fits: it hosts the ladder next to the model endpoint, and the free model access gives you a real endpoint to protect. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Record three numbers. Attempts per request. Delay per attempt. Final status. Compare them to the mock run. Expect a surprise. Real endpoints drift from mocks. The Retry-After value may be missing. The timeout may fire first. Adjust the ladder, then rerun the mock to confirm the change. If the real endpoint returns 200 with a broken payload, the ladder is not the right tool. That is a validation problem, not a transport problem.

What the final flow looks like

The full flow is four layers. Classification decides if a retry helps. The ladder spaces the attempts. The breaker stops the storm. The cache deduplicates the results. Each layer is testable in isolation. Together they turn a flaky endpoint into a predictable one. The cost is latency. The benefit is surviving a failure window without manual intervention.

Limitations

Retries do not create quota. If the 429 means "allowance spent", backoff just delays the failure. Plan the quota separately. The breaker adds latency to every request. It pays off only when many requests share one endpoint. A single script with one user may never need it. LLM retries are not idempotent. The cache reduces duplicates but cannot guarantee identical completions. Validate the retried output before you trust it. Timeouts are guesses. Measure the p95 latency first. A 10-second timeout on a 30-second endpoint fails every time. The numbers in this tutorial are starting points. Your endpoint has its own limits. Measure before you tune.

Who should not use this

Real-time UIs that need a response in under a second. Systems that mutate external state on every call. Teams with a paid SLA who should use the provider's native retry SDK. If you cannot tolerate a delayed response, a retry ladder is the wrong layer.

Your turn: change the SCRIPT list and watch the ladder adapt. Then point it at a real endpoint and graph the attempt counts. A 429 storm is a teacher. The ladder just makes the lesson repeatable.