Every benchmark announcement on the front page this week has the same shape: a score, a chart, a happy path. Almost nobody publishes the boring trace — the 429, the 500, the retry that fired twice and created two records.
Here is my core claim. Your retry code is not tested until you have fed it failures on purpose, on a schedule you control. Reading the code and nodding is not testing. Hitting a flaky free endpoint and "seeing it work" is not testing either.
So let's build the failure source first, then the client, then prove the whole thing against a real endpoint. Five stages. Each one has a verification step, because an unverified stage is just a story.
What you need
- Python 3.11+ and nothing else for stages 1–3. Standard library only, no keys, no SDK.
- A terminal and roughly 20 minutes.
- For stage 4, a real endpoint and somewhere to run the harness. I'll come back to that.
No Docker, no pytest required. I'll use plain assert so the checks stay copy-pasteable.
Stage 1 — Build a scripted failure server
Stop waiting for a real API to misbehave. Write a server that fails exactly when you tell it to.
# injector.py — scripted failure server, stdlib only
import json, sys, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
SCRIPT = json.load(open(sys.argv[1])) # list of responses, consumed in order
STATE = {"i": 0, "keys": []}
class H(BaseHTTPRequestHandler):
def do_POST(self):
body = self.rfile.read(int(self.headers.get("content-length", 0)))
req = json.loads(body or b"{}")
key = self.headers.get("Idempotency-Key")
if key and key in STATE["keys"]: # duplicate retry already accepted
return self._send(200, {"dedup": True})
step = SCRIPT[min(STATE["i"], len(SCRIPT) - 1)]
STATE["i"] += 1
if step.get("delay_ms"):
time.sleep(step["delay_ms"] / 1000)
if step["status"] == 200 and key:
STATE["keys"].append(key)
headers = {}
if "retry_after" in step:
headers["Retry-After"] = str(step["retry_after"])
self._send(step["status"], {"attempt": STATE["i"], "echo": req}, headers)
def _send(self, status, payload, headers=None):
raw = json.dumps(payload).encode()
self.send_response(status)
self.send_header("content-type", "application/json")
self.send_header("content-length", str(len(raw)))
for k, v in (headers or {}).items():
self.send_header(k, v)
self.end_headers()
self.wfile.write(raw)
def log_message(self, *a):
pass
ThreadingHTTPServer(("127.0.0.1", 8787), H).serve_forever()
Now the script it plays: rate limited, then a server error, then success.
[
{"status": 429, "retry_after": 1},
{"status": 500},
{"status": 200}
]
Verify stage 1. Run the server and poke it three times. You should get three different codes in a fixed order.
python injector.py schedule.json &
for i in 1 2 3; do
curl -s -o /dev/null -w "%{http_code}\n" -X POST http://127.0.0.1:8787/v1/chat \
-H 'content-type: application/json' -d '{"prompt":"hi"}'
done # expected: 429, 500, 200
If the order drifts, your server is sharing state across requests it shouldn't. Fix that before moving on.
Stage 2 — A client with a deadline, not just a counter
Most retry loops I read have a max-attempts counter and no wall-clock budget. That is how a client burns 40 seconds retrying something the caller gave up on 30 seconds ago.
# client.py
import json, random, time, urllib.error, urllib.request
RETRYABLE = {429, 500, 502, 503, 504}
def call(payload, key, base_url, deadline_s=8.0, max_attempts=5, base=0.25, cap=2.0):
t0, trace = time.monotonic(), []
for attempt in range(1, max_attempts + 1):
remaining = deadline_s - (time.monotonic() - t0)
if remaining <= 0:
trace.append((attempt, "deadline"))
break
req = urllib.request.Request(
base_url + "/v1/chat",
data=json.dumps(payload).encode(),
headers={"content-type": "application/json", "Idempotency-Key": key},
method="POST")
try:
with urllib.request.urlopen(req, timeout=remaining) as r:
trace.append((attempt, r.status))
return json.load(r), trace
except urllib.error.HTTPError as e:
trace.append((attempt, e.code))
if e.code not in RETRYABLE or attempt == max_attempts:
raise
ra = e.headers.get("Retry-After")
wait = float(ra) if ra else min(cap, base * 2 ** (attempt - 1))
wait += random.uniform(0, wait * 0.25) # decorrelate fleet retries
except (TimeoutError, urllib.error.URLError):
trace.append((attempt, "transport"))
wait = min(cap, base * 2 ** (attempt - 1))
wait += random.uniform(0, wait * 0.25)
if time.monotonic() - t0 + wait > deadline_s:
trace.append((attempt + 1, "deadline"))
break
time.sleep(wait)
raise TimeoutError(trace)
Three decisions worth defending. Retry-After wins over your own backoff, always — the server knows something you don't. Jitter is not optional, because synchronized retries turn one blip into a second outage. And a timeout after the request was sent is not a safe "nothing happened"; treat it as unknown.
Verify stage 2.
python - <<'PY'
from client import call
body, trace = call({"prompt": "hi"}, "key-abc", "http://127.0.0.1:8787")
print(trace)
assert trace == [(1, 429), (2, 500), (3, 200)], trace
assert body["attempt"] == 3
print("ok")
PY
Reset the injector between runs, since the script is consumed in order. If your trace shows four attempts here, you are double-calling somewhere — usually a finally block.
Stage 3 — Prove retries don't duplicate the work
Here is the failure nobody tests: the request succeeded server-side, the response never arrived, and your client retried. Without an idempotency key you now have two summaries, two charges, or two rows.
Change the schedule to a slow success:
[{"status": 200, "delay_ms": 1500}]
Verify stage 3. Force the client's timeout below the server's delay, and confirm the retry is recognized as a duplicate instead of creating new work.
python - <<'PY'
from client import call
body, trace = call({"prompt": "hi"}, "key-abc", "http://127.0.0.1:8787", deadline_s=4.0)
print(trace, body)
assert trace[0][1] == "transport" # first attempt timed out
assert body.get("dedup") is True # second attempt saw the same key
print("no duplicate accepted output")
PY
This is the only stage that changes correctness rather than latency. Skip it and you have a retry loop that is fast and wrong.
Stage 4 — Point the same harness at a real endpoint
A scripted injector proves your control flow. It does not prove the real endpoint's error shapes. So parameterize the base URL and run the identical client against a live API with the injector's schema in mind.
Two guardrails. Keep concurrency at 1–2 while you are learning the error shapes. And honor Retry-After — retrying is for recovering from transient faults, never for getting around a limit the provider set deliberately.
This is where a free tier is genuinely useful as a test surface, not as a production dependency. MonkeyCode exposes free model access and, per its operator, a free server option — that combination is enough to run this harness end to end without a billing account, and the server option means your test traffic crosses a real network hop instead of loopback, which is where timeouts actually happen.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
I'm not going to quote a token number or a machine spec here, because quotas and limits change and a stale number in a tutorial is worse than no number. Check the project's current terms before you plan capacity around them.
Verify stage 4. Capture the real trace and assert only what you know is true: attempts to success, whether any Retry-After arrived, and whether the deadline was ever breached.
BASE=https://your-endpoint.example/v1 python - <<'PY'
import os
from client import call
_, trace = call({"prompt": "hi"}, "probe-1", os.environ["BASE"], deadline_s=10.0)
print(trace)
assert trace[-1][1] == 200, trace
PY
One clean run proves nothing. Run it 20 times in a loop, or you have sampled exactly one weather condition.
Stage 5 — Read the trace like a decision table
A trace is only useful if it changes what you ship.
| Observation over 20 runs | Interpretation | Action |
|---|---|---|
| 20/20 first-attempt 200s | Clean path, untested path | Keep injector tests in CI; you have no evidence about failure behavior |
429 with Retry-After on most runs |
You are above the tier's comfortable rate | Lower concurrency; do not shorten backoff |
transport timeouts recurring at the same duration |
Timeout is shorter than real latency | Raise timeout, not attempt count |
dedup: true appearing often |
Retries are landing after success | Idempotency is load-bearing; keep the key forever |
deadline in the trace |
Budget math is wrong | Cut max_attempts or raise deadline_s; never both silently |
That last row is the one I care about most. A client that gives up on schedule is honest. A client that retries past its caller's patience is just adding load to someone else's incident.
What this does not tell you
It is a failure-model harness, not a load test. It says nothing about output quality, throughput ceilings, or whether the model is any good at your task.
It also assumes the endpoint honors idempotency keys. If it does not, you need a client-side store keyed on request hash, and you need to accept that the duplicate may already exist server-side.
Skip this approach if you are streaming tokens — a deadline-plus-retry model doesn't map cleanly onto a partially consumed stream — or if you need exactly-once semantics without an idempotency-capable API, or if you need a capacity plan rather than a correctness plan. Free tiers are fine for learning failure shapes; they are not a foundation for an SLA.
Do one thing today
If you only take one step, take stage 1. A forty-line injector that returns 429 and 500 on command is the cheapest test infrastructure you will ever write, and it will find the retry bug in your code before a user does.
What does your trace look like — three attempts and a clean exit, or a deadline you didn't know you were hitting?
Top comments (0)