Stop Retrying Blind: A Failure-Bucket FAQ for Free Model Endpoints

The model answers in 250 ms. Your chat still feels broken.

Why? The layer between your code and the endpoint. Timeouts, status codes, retries. Everyone reviews the prompt. Nobody reviews the failure path.

I used a free model endpoint on MonkeyCode's free server as the stand-in for this post. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness is endpoint-agnostic. The buckets are the point.

Bucket first, backoff second

Everyone adds exponential backoff. Almost nobody tags failures first.

A bucket is one log line: TIMEOUT, HTTP_429, HTTP_503, HTTP_5XX, CONN_RESET, BAD_RESPONSE, OK. Bucketing before retrying shows two things: which failures are rare, and which ones your retries are making worse.

This is the harness I run before I touch retry code.

# failure_buckets.py — Python 3.10+
# Measure these rates on your own endpoint. Never copy them from a post.
PREFILL_TOK_PER_SEC = 250.0   # tokens/sec reading your prompt
DECODE_TOK_PER_SEC  = 45.0    # tokens/sec writing the answer
OVERHEAD_SECONDS    = 8.0     # connect + TLS + serialization
SLACK               = 1.3
MAX_ATTEMPTS        = 3

def timeout_budget(prompt_tokens: int, output_tokens: int) -> float:
    return SLACK * (
        OVERHEAD_SECONDS
        + prompt_tokens / PREFILL_TOK_PER_SEC
        + output_tokens / DECODE_TOK_PER_SEC
    )

looks_sane = lambda text: bool(text.strip())  # set your own validity check

def bucket(elapsed: float, budget: float, status: int | None, text: str) -> str:
    if status == 200 and looks_sane(text):
        return "OK"
    if status == 200:
        return "BAD_RESPONSE"      # 200 with garbage is still garbage
    if status == 429:
        return "HTTP_429"
    if status == 503:
        return "HTTP_503"
    if status and 500 <= status < 600:
        return "HTTP_5XX"
    if elapsed >= budget:
        return "TIMEOUT"
    return "CONN_RESET"

def decide(kind: str, attempt: int, retry_after: str | None) -> str:
    if kind == "OK":
        return "return"
    if kind == "BAD_RESPONSE":
        return "give up"           # same input, same garbage
    if kind == "TIMEOUT":
        return "retry only if the call has no side effects"
    if kind == "HTTP_429" and retry_after:
        return f"sleep exactly {retry_after}s"
    if kind == "HTTP_429":
        return f"sleep {min(2 ** attempt, 8)}s + jitter"
    if kind in ("HTTP_503", "CONN_RESET", "HTTP_5XX"):
        return "retry after 1s" if attempt < MAX_ATTEMPTS else "give up"
    return "give up"

The runtime loop is intentionally a sketch. Wire it to your own HTTP client.

deadline = time.monotonic() + 25.0   # total budget: the user is waiting
attempt = 0
while time.monotonic() < deadline and attempt < MAX_ATTEMPTS:
    budget = timeout_budget(prompt_tokens=1200, output_tokens=300)
    kind, retry_after = post_with_timeout(prompt, budget)  # your client
    action = decide(kind, attempt, retry_after)
    # ... apply action, log bucket, increment attempt ...

The script is deliberately boring. It waits, buckets, and picks an action from a table. The five beliefs below are why that table exists.

Belief 1: "A fixed 30-second timeout is fair"

A fixed timeout ignores the shape of the request. Reading 4,000 input tokens and writing 50 output tokens are different jobs.

A token-aware budget is the honest version:

timeout = overhead + prompt_tokens / prefill_rate + output_tokens / decode_rate

Measure the two rates locally. Time five requests, take the median, and update the constants. If your timeout is constant, your timeout is wrong. It must scale with the tokens you asked for.

Belief 2: "A timeout means the request never ran"

A timeout closes your socket. It does not cancel the server's work. The generation may still finish, spend quota, and fire follow-ups you never see.

Retrying in that state runs the same work twice. Twice is fine for a text echo. Twice is a bug when the call triggers a webhook, a write, or a payment.

Treat a timeout as "unknown." Retry only when the call is side-effect-free. If it is not, surface the ambiguity to the user instead of stacking duplicate work.

Belief 3: "A 429 means I should stop asking"

Sometimes it does. When Retry-After is present, the server gave you the number. Sleep exactly that long. Sleeping longer just adds latency.

A bare 429 is a different animal. On a shared free queue it often means contention, not a hard ban. Short jittered backoff keeps you in the game. Stopping for a minute takes you out of it.

Decision: honor Retry-After when you see it. Otherwise back off for one or two seconds and retry.

Belief 4: "More retries mean more reliability"

On a shared free server, every retry is a new arrival at the back of the same queue. Retrying under load does not rescue the request. It feeds the queue.

The herd is real: ten clients, three retries each, and the endpoint sees thirty new arrivals. The failure you measured multiplies into the failure you caused.

A deadline fixes this. Cap the total retry time (25 seconds in the harness) and cap attempts. Then let decide() pick actions from one table:

Bucket	Action
OK	return
TIMEOUT	retry only if side-effect-free
HTTP_429	honor Retry-After or short jitter
HTTP_503	short retry, then give up
CONN_RESET	short retry, then give up
HTTP_5XX	backoff, then give up
BAD_RESPONSE	never retry — same garbage

Reliability comes from the deadline, not from the retry count.

Belief 5: "Parallel calls make it faster"

A free endpoint usually shares a worker pool among many users. Parallel requests do not create more workers. They multiply contention.

Probe the knee before you believe in parallelism.

from concurrent.futures import ThreadPoolExecutor
import time, requests

URL = "https://your-endpoint.example"  # set me

def first_byte_ms(url: str) -> float:
    t0 = time.perf_counter()
    with requests.post(url, json={"prompt": "Say hi"}, stream=True, timeout=30) as r:
        next(r.iter_content(1))          # first byte after queue + prefill
    return (time.perf_counter() - t0) * 1000

for level in (1, 2, 4, 8, 16):
    with ThreadPoolExecutor(level) as pool:
        samples = list(pool.map(lambda _: first_byte_ms(URL), range(20)))
    print(level, round(sum(samples) / len(samples), 1), "ms")

Plot the medians. The knee is your ceiling. Stay far below it and cap your thread pool.

Who should skip this harness

This approach has an off switch.

If you run your own server, fix the server. Routing around a server you control is wasted motion. If a paid SLA guarantees your numbers, fail fast and let the user retry. If you make fewer than about a thousand calls a day, keep one constant timeout and move on.

One more limit: buckets describe behavior, not root cause. A wave of HTTP_503 tells you how much load. It does not tell you which setting caused it. Pair the buckets with provider docs before you speculate.

The buckets are the evidence

The five beliefs above are hypotheses. Bucket counts are evidence.

Run the harness for one afternoon before you add another retry. Keep the ratios — 503 share versus 429 share versus timeout share — and write the policy that fits them. If you need a free endpoint to practice on, MonkeyCode's free server works. Any free HTTP model API works too. The harness does not care which one you choose.