Your retry loop is probably the most expensive thing you never benchmarked.
You just hit a 429. What now? Most people retry. Some add exponential backoff. Almost nobody reads the header that explains why.
I spent a weekend classifying failures on a free model endpoint. I ran the probe from MonkeyCode's free server option, against MonkeyCode's free model access. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The script itself is endpoint-agnostic. Point it at any OpenAI-compatible chat API and watch your failures stop being opinions.
Myth 1: "A 429 means I burned my quota"
A 429 is a family of answers, not a single verdict. It can mean your quota is gone. It can also mean the queue is full for a second. Retry-After is the tiebreaker. Present and large? Stop retrying. Absent or single-digit? That's queue pressure, not a quota wall.
Assuming every 429 is a quota makes you wait when you should retry. Assuming the opposite makes you hammer a wall you cannot pass.
Myth 2: "Exponential backoff is the professional choice"
Exponential backoff is the textbook answer. It is also a great way to synchronize a thundering herd. Every client computes the same sleep, then slams back at the same moment. Marc Brooker's classic post on exponential backoff and jitter explains why shared systems need jitter. A jittered fixed delay often beats pure exponential backoff on a shared queue.
For a chat endpoint, start with random.uniform(0.2, 0.8) seconds. Measure. Tune later.
Myth 3: "HTTP 200 means the request worked"
A 200 means the gateway sent you a body. It does not mean the body is valid. You can get empty choices. You can get a finish_reason of length instead of stop. That's a truncated generation wearing a success costume. Your logs show a clean 200. Your users see a half answer.
Validate the payload, not just the status code.
Myth 4: "My 45-second timeout tells me the server is slow"
Your timeout measures your whole path. DNS. TCP. TLS handshake. Proxy. Queue position. On a queued free endpoint, the clock starts before the model does. Twenty seconds in the queue plus ten seconds of generation eats a 30-second budget. The request was healthy the entire time.
Separate queue time from generation time before you tune anything.
Myth 5: "Retries are free because the first call failed anyway"
Each retry re-enters the same queue. It competes with your other in-flight calls. It also extends congestion for everyone sharing that endpoint. The money cost is near zero. The latency cost is not.
The only honest price for a retry is measured: how many attempts does a successful call need? That number is your real free-tier tax.
The probe: classify before you configure
Stop guessing about failure classes. This probe sends a short, cheap request in a loop. It records the status, the Retry-After header, the latency, and the finish_reason. Then it classifies each failure and applies a jittered retry.
Python 3 only. No third-party packages.
# retry_probe.py
# Classify failures on any OpenAI-compatible chat endpoint.
import json
import random
import time
import urllib.error
import urllib.request
BASE_URL = 'https://your-endpoint.example/v1/chat/completions'
API_KEY = 'sk-your-key' # use an env var; never commit
def call_once(payload):
req = urllib.request.Request(
BASE_URL,
data=json.dumps(payload).encode('utf-8'),
headers={
'Authorization': 'Bearer ' + API_KEY,
'Content-Type': 'application/json',
},
)
start = time.perf_counter()
try:
with urllib.request.urlopen(req, timeout=45) as resp:
body = json.loads(resp.read().decode('utf-8'))
choice = body['choices'][0]
return {
'status': resp.status,
'retry_after': resp.headers.get('Retry-After'),
'latency': time.perf_counter() - start,
'finish': choice.get('finish_reason'),
'text_len': len(choice['message']['content'] or ''),
}
except urllib.error.HTTPError as e:
return {
'status': e.code,
'retry_after': e.headers.get('Retry-After'),
'latency': time.perf_counter() - start,
'finish': None,
'text_len': 0,
}
except urllib.error.URLError:
return {
'status': None,
'retry_after': None,
'latency': time.perf_counter() - start,
'finish': None,
'text_len': 0,
}
def classify(r):
if r['status'] is None:
return 'timeout'
if r['status'] == 429:
after = r['retry_after']
if after and after.isdigit() and int(after) >= 30:
return 'quota'
return 'queue'
if r['status'] >= 500:
return 'server'
if r['status'] == 200 and r['finish'] != 'stop':
return 'truncated'
if r['status'] == 200:
return 'ok'
return 'config'
def run(payload, total=60, max_attempts=4):
counts = {}
success_by_attempt = [0] * (max_attempts + 1)
for _ in range(total):
for attempt in range(1, max_attempts + 1):
result = call_once(payload)
cls = classify(result)
if cls == 'ok':
counts['ok'] = counts.get('ok', 0) + 1
success_by_attempt[attempt] += 1
break
if cls in ('quota', 'config', 'truncated'):
counts[cls] = counts.get(cls, 0) + 1
break
counts[cls + '_retry'] = counts.get(cls + '_retry', 0) + 1
time.sleep(random.uniform(0.2, 0.8) * attempt)
return counts, success_by_attempt
if __name__ == '__main__':
payload = {
'model': 'your-model-id',
'messages': [{'role': 'user', 'content': 'Say ok once.'}],
'max_tokens': 16,
}
counts, attempts = run(payload)
print('classification:', counts)
print('ok by attempt:', attempts[1:])
Run it with python retry_probe.py. Let it finish 60 calls. Then read the two printed lines.
How to read the output
-
classificationtells you which myth dominates your endpoint. -
ok by attempttells you the real retry tax. Most wins on attempt 1? Your backoff is irrelevant. Most wins on attempt 3? You pay queue penalties on every request.
I left this probe running on MonkeyCode's free server so my laptop and my production quota stayed out of the experiment. The same script works on any machine. Just respect the endpoint's rate limits while you probe. If your endpoint is slow, lower total to 20 first.
The decision table
Translate what you see into what you do.
| Signal you see | Likely meaning | Action |
|---|---|---|
429 + Retry-After >= 30 |
quota or plan cap | stop; check your quota endpoint |
429 + no Retry-After
|
queue saturation | retry with jitter, 0.2-0.8s |
| 429 + 5xx cluster | server-wide issue | back off 5s+, log, alert |
200 + finish_reason: length
|
truncation | raise max_tokens or shrink the prompt |
200 + empty choices
|
request shape issue | fix the payload; retrying won't help |
| 5xx | upstream failure | retry with jitter, cap at 3 attempts |
| timeout | queue + network mix | split queue time from generation time |
The corrected mental model
- A 429 is a queue signal until the headers prove it's a quota.
- A 200 is a contract with the body, not with the gateway.
- Retry policy is a distribution, not a switch. Measure success by attempt number.
- Exponential backoff without jitter is a coordination bug.
- The free tier doesn't charge money for retries. It charges queue position.
Limitations and who should skip this
This probe assumes a synchronous, OpenAI-compatible chat endpoint. Streaming and tool-calling paths need extra checks for partial frames and mid-stream errors. Small sample sizes mislead; run at least 60 calls spread across different minutes. And the probe deliberately generates 429s, so it consumes queue capacity. Don't run it against a quota you need for production traffic.
Skip this approach when your endpoint enforces hard per-minute quotas and every retry burns credits. Skip it when your application has strict latency SLAs and can't afford in-line retries — push the work to a background job instead. And never paste a real API key into a public gist. Read it from an environment variable.
If your 429s are still a mystery, this is the cheapest experiment you'll run this week. Run the probe, then tell me what your decision table looks like.
Top comments (0)