429 Is Not a Failure: Five Retry Fallacies You Can Measure on a Free Model Server

Last week, my retry loop ate several minutes of wall time.

Every call returned 429. My worker waited one second, retried, and hit the throttle again. The model server never crashed. The queue never cleared. My own client was pressuring its own tickets.

If you build on a free model server, you have repeated the same chants. "Just retry it." "Set a shorter timeout." "More concurrency fixes it." All three are retry fallacies. This FAQ shows you how to measure each one instead of trusting the rumor.

I put the probe together while testing the free model access and the optional free server MonkeyCode provides. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The probe is not a product feature. It is a plain Python script and a little measurement discipline.

The Probe

Save this as free_flow_probe.py.

import json
import sys
import time
import urllib.error
import urllib.request
from concurrent.futures import ThreadPoolExecutor

ENDPOINT = sys.argv[1] if len(sys.argv) > 1 else "https://your-gateway/v1/completions"
CONCURRENCY = int(sys.argv[2]) if len(sys.argv) > 2 else 8
RETRY_WAIT = float(sys.argv[3]) if len(sys.argv) > 3 else 0.0

PAYLOAD = {"prompt": "Count to three.", "max_tokens": 8}


def call_once():
    body = json.dumps(PAYLOAD).encode()
    req = urllib.request.Request(ENDPOINT, data=body, method="POST")
    req.add_header("Content-Type", "application/json")
    with urllib.request.urlopen(req, timeout=60) as resp:
        return resp.status


def worker(_):
    attempts = 0
    t0 = time.monotonic()
    while True:
        attempts += 1
        try:
            status = call_once()
            return attempts, status, time.monotonic() - t0
        except urllib.error.HTTPError as err:
            if err.code == 429 and RETRY_WAIT > 0:
                time.sleep(RETRY_WAIT)
                continue
            return attempts, err.code, time.monotonic() - t0
        except Exception as err:
            return attempts, type(err).__name__, time.monotonic() - t0


with ThreadPoolExecutor(max_workers=CONCURRENCY) as pool:
    results = list(pool.map(worker, range(CONCURRENCY)))

total_time = max(r[2] for r in results)
print(f"jobs={len(results)} successful={sum(1 for r in results if r[1] == 200)}")
print(f"attempts={sum(r[0] for r in results)} wall_time={total_time:.2f}s")

Add your auth header inside call_once if your gateway requires one. The script sends CONCURRENCY tiny completion requests to any OpenAI-compatible endpoint. It counts three numbers:

• attempts — how many times your client hit the endpoint
• wall_time — how long the whole batch took
• successful — how many jobs returned a 200

Run it twice.

python free_flow_probe.py https://your-gateway/v1/completions 8 0
python free_flow_probe.py https://your-gateway/v1/completions 8 2

The first run retries instantly. The second waits two seconds after every 429.

Sample output (illustrative only — your gateway will differ):

# retry_wait=0
jobs=8 successful=8 attempts=23 wall_time=31.40s

# retry_wait=2
jobs=8 successful=8 attempts=8 wall_time=20.05s

Now, the fallacies.

Fallacy 1 — "429 means my request failed"

The HTTP spec calls 429 "Too Many Requests" (RFC 9110). Too many for this moment, not for forever.

On a shared free server, a 429 is a queue signal. It means "you will be served, but not right now." Treat it like a full line at the checkout, not a fatal error. The probe shows the pattern: most jobs eventually succeed. The batch is slower, not dead.

Delete this assumption: 429 = failure. Replace it with: 429 = backpressure schedule.

Fallacy 2 — "Retrying instantly is faster than backing off"

An instant retry re-enters the same queue. You do not jump the line. You buy a new ticket and wait again.

Every immediate retry multiplies the shared load. With RETRY_WAIT=0, attempts climb far above jobs. With RETRY_WAIT=2, attempts stay close to jobs. Wall time stays roughly the same. The only difference is the noise you add to everyone else's queue.

Instant retries are the free-tier version of shouting "are we there yet?"

Fallacy 3 — "A shorter timeout protects me from slow servers"

A timeout does not cancel the server's work.

Most gateways finish the completion and then drop the response into the void. Your client has already left. Your retry starts the same computation again. You consume the request budget twice for one result.

Measure wall_time before shrinking your timeout. If the endpoint is congested, the fix is backing off, not giving up earlier.

Fallacy 4 — "More concurrency means more throughput"

Free tiers do not behave like CPU cores. They behave like checkout queues.

Concurrency inserts eight tickets into the same lane. Then retries add eight more. Your concurrency becomes a multiplier on queue length, not on throughput. Doubling concurrency rarely doubles throughput. It usually doubles attempts.

Run the probe with 4 workers, then 16. Watch wall_time climb while successful stays flat. That is the shape of a shared queue, not a shared machine.

Fallacy 5 — "Retry-After is decorative"

The Retry-After header (MDN) is the only precise wait signal the server gives you.

Free gateways rarely send it. When they do, use the value verbatim. When they do not, use exponential backoff with jitter: 1s, 2s, 4s, plus randomness. Do not reuse one policy for every error. Only 429 and 5xx deserve retries. A 400 will not heal with patience.

A corrected client loop

Pseudo-code sketch — adapt it to your client:

attempt = 0
while True:
    try:
        return call_model(prompt)
    except HTTPError as err:
        if err.code == 429:
            wait = retry_after(err) or backoff(attempt)
            sleep(wait)
            attempt += 1
            continue
        if err.code >= 500:
            sleep(backoff(attempt))
            attempt += 1
            continue
        raise

This separates 429 from crashes. Both use backoff. But 429 respects the server's schedule, while 5xx respects your own.

Where this breaks

• The probe measures network-visible backpressure, not KV-cache hits, sampler behavior, or internal model queues.
• The probe is designed for small loads. Do not use it to stress a shared endpoint. You will burn your own quota while annoying other tenants.
• If your endpoint has an SLA, ignore everything above. A committed endpoint gives you guarantees. A free tier gives you a queue position.

Who should not use this workflow

• Production code behind a hard function timeout. Retries on a free endpoint will race that timeout and often lose.
• Real-time interfaces that need stable latency. Free tiers are bursty. Budget for the worst p95, not the morning average.
• Teams promising "it just works" in production. Free tiers validate ideas. They do not deliver contracts.

Run it

Run the probe against your own gateway endpoint. Twice: one instant-retry run, one backed-off run. Then decide whether your retry policy is an accelerator or a brake.