My batch job reported a 14% error rate, and I spent the first evening rewriting retry logic that was never the problem. The number came from a single timeout=30 on an HTTP client that could not tell me whether the connection failed, the server went quiet, or generation simply outran my patience. So I ran the same prompt mix for 48 hours against a free model endpoint on a free server, and I logged every attempt instead of only the failures. Here is what broke, what I changed, and what I would repeat.
What I actually ran
The plan was deliberately boring, because a clever harness would have hidden the thing I wanted to see. I kept one prompt mix, one machine, and one network, and I wrote every attempt to an append-only JSONL file before touching any aggregation code.
- Day one: no streaming, one flat 30-second timeout, naive retry on any exception.
- Day two: split connect/read/write timeouts, streaming enabled, and a retry classifier in front of the loop.
- Both days: the same prompts in the same order, plus a deliberate warm-up call after long idle gaps.
- Analysis: tail percentiles and a waste ratio, computed from the ledger rather than from memory.
Was the 14% a property of the endpoint, my client, or my prompt mix? Without a per-attempt record, that question is unanswerable, and unanswerable questions turn into folklore fast.
The bug was in my error report, not my code
When I grepped the day-one ledger, almost every "failure" carried the same label, because a flat timeout collapses distinct events into one exception type. A connection that never opened and a generation that produced tokens for 40 seconds looked identical to my retry loop. That is why the retry storm was invisible: I was retrying a slow-but-healthy request as if it were a dead socket.
And retries are not free, even on a free tier. Each duplicate attempt spends wall clock, spends rate-limit headroom you may not be able to see, and pollutes the very metric you are trying to fix.
Artifact 1: a per-attempt ledger you can grep
This is the wrapper I wish I had written on hour one. It records a prompt hash instead of the prompt, splits the timeout budget, and classifies the outcome before deciding whether to retry.
import hashlib, json, random, time, uuid
from pathlib import Path
import httpx
LEDGER = Path("attempts.jsonl")
NON_RETRYABLE = {400, 401, 403, 404, 422}
def prompt_hash(prompt: str) -> str:
return hashlib.sha256(prompt.encode("utf-8")).hexdigest()[:16]
def log(record: dict) -> None:
with LEDGER.open("a", encoding="utf-8") as fh:
fh.write(json.dumps(record, sort_keys=True) + "\n")
def call(prompt: str, url: str, headers: dict, max_attempts: int = 4):
key = prompt_hash(prompt)
for attempt in range(1, max_attempts + 1):
started = time.monotonic()
rec = {
"key": key,
"attempt": attempt,
"request_id": str(uuid.uuid4()),
"started_at": time.time(),
"outcome": None,
}
timeout = httpx.Timeout(connect=5.0, read=60.0, write=10.0, pool=10.0)
try:
with httpx.Client(timeout=timeout) as client:
with client.stream("POST", url, json={"prompt": prompt}, headers=headers) as r:
rec["status"] = r.status_code
first_byte = None
chunks = []
for chunk in r.iter_bytes():
if first_byte is None:
first_byte = time.monotonic()
rec["first_byte_ms"] = round((first_byte - started) * 1000)
chunks.append(chunk)
r.raise_for_status()
rec["outcome"] = "ok"
rec["body"] = b"".join(chunks).decode("utf-8", "replace")
except httpx.TimeoutException as exc:
rec["outcome"] = "timeout"
rec["detail"] = type(exc).__name__
except httpx.HTTPStatusError as exc:
rec["outcome"] = "http_error"
rec["detail"] = exc.response.status_code
finally:
rec["total_ms"] = round((time.monotonic() - started) * 1000)
log(rec)
if rec["outcome"] == "ok":
return rec
if rec["outcome"] == "http_error" and rec["detail"] in NON_RETRYABLE:
raise RuntimeError(f"non-retryable {rec['detail']} for {key}")
time.sleep(min(30.0, 2 ** attempt) + random.uniform(0, 1))
raise RuntimeError(f"gave up after {max_attempts} attempts for {key}")
Two details matter more than they look. first_byte_ms is the field that tells you whether you were waiting on the network or on generation, and the key is what later lets you count duplicates. If your client library cannot stream, log connect_ms separately and accept a coarser story.
Artifact 2: classify the failure before you retry
Before day two I wrote down, on paper, what each log line should mean and what the client should do about it. This table is the whole reason the error report changed shape.
| What the ledger says | What it usually means | Retry? |
|---|---|---|
timeout / ConnectTimeout
|
Socket, DNS or TLS never established | Yes, short backoff, cap at 2 |
timeout / ReadTimeout, no first_byte_ms
|
Accepted, but no token produced yet (cold start, queueing) | Yes, with a longer read window |
timeout / ReadTimeout, with first_byte_ms
|
Generation genuinely slower than your window | Not with the same window |
http_error 429 |
Rate limited | Yes, honour Retry-After
|
http_error 5xx |
Server side, often transient | Yes, jittered |
http_error 400/401/422 |
Your request is wrong | No, fix the request |
Is your retry loop currently able to distinguish row two from row three? If not, you are mixing a warm-up problem with a generation-speed problem, and neither fix will look like it worked.
Artifact 3: turn the ledger into a tail report
Percentiles from a run of a few hundred attempts are noisy, so treat them as a shape, not a spec. What I wanted was the gap between the median and the tail, plus how much work was spent on duplicates.
import json
from collections import Counter
from pathlib import Path
rows = [json.loads(line) for line in Path("attempts.jsonl").read_text().splitlines() if line.strip()]
ok = [r for r in rows if r["outcome"] == "ok"]
def pct(values, p):
if not values:
return None
values = sorted(values)
i = min(len(values) - 1, int(round((p / 100) * (len(values) - 1))))
return values[i]
totals = [r["total_ms"] for r in ok]
firsts = [r["first_byte_ms"] for r in ok if "first_byte_ms" in r]
print("ok attempts :", len(ok))
print("total p50/p95 :", pct(totals, 50), pct(totals, 95))
print("first-byte p50/95:", pct(firsts, 50), pct(firsts, 95))
print("outcomes :", Counter((r["outcome"], r.get("detail")) for r in rows))
keys = {r["key"] for r in rows}
print("attempts/request :", round(len(rows) / max(1, len(keys)), 2))
slow = [r for r in ok if r["total_ms"] > 30_000]
print("slow successes :", len(slow), "(invisible to a flat timeout report)")
For a quick look without Python, this one-liner was enough during the run:
jq -r 'select(.outcome != "ok") | [.outcome, (.detail|tostring)] | @tsv' attempts.jsonl \
| sort | uniq -c | sort -rn
That command is the difference between "it failed a lot" and "most of my failures were read timeouts with no first byte". One is a mood; the other is a bug report.
What I changed on day two
- Split the timeout budget: connect 5s, write 10s, read 60s, with an overall wall-clock cap per request.
- Streamed responses so a first byte becomes an observable event instead of a guess.
- Classified before retrying, and refused to retry 4xx client errors at all.
- Added full jitter to the backoff, capped attempts at four, and logged the give-up.
- Separated warm-up calls from the measured set, so idle gaps stopped poisoning the failure rate.
None of these are exotic, and that is the point. The expensive part was not the fix; it was discovering that a single number had been standing in for three different phenomena.
The free-server detail that reshaped the numbers
A free server option usually means you are sharing something, and shared things can be reclaimed or idled in ways a paid box is not. I did not assume any particular policy, because free-tier behaviour changes with plan and region and is documented by the provider, not by me. I read the current docs first, then designed the harness so that the first request after a quiet period was tagged as warm-up rather than counted as a failure.
Does your batch job have any concept of a warm-up request? If the answer is no, your first error of the morning may be a scheduling artefact that you have been debugging as a code defect.
Where MonkeyCode fit
I used MonkeyCode's free model access and its free server option to host the harness and run the calls for two days, which kept the experiment cheap enough to repeat when the first version was wrong. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Product details such as quotas, idle behaviour and available models are exactly the things that change, so check the current documentation rather than trusting a blog post, including this one.
If you want to rerun these field notes with your own prompt mix, free model access plus a free server is sufficient to generate the ledger; the numbers below are yours to earn, not mine to guess.
Limitations, and who should skip this
- This was one window, one machine, one prompt mix. It is a debugging workflow, not a benchmark of anything.
- Prompt hashes keep the ledger shareable, but you still should not log response bodies from real user data.
- Free tiers are shared and can change; do not build a latency SLO or a cost model on top of one.
- If your workload is a handful of calls per day, the ledger is overhead. Read the provider docs and move on.
- If you need hard per-request cost accounting, this approach will not give it to you, because free access does not emit a price per call.
What I would repeat
Writing the ledger before writing the retry logic, keeping the prompt mix fixed across both days, and reading the timeout table out loud to myself before trusting any dashboard. The thing I would not repeat is the flat 30-second timeout, which felt like a simplification and behaved like a lie. Next time I would also run a small, known-good request first each morning to fingerprint the day before blaming the code.
What is your error report actually measuring: the endpoint, your network, or your own patience?
Top comments (0)