Your request timed out. What's your first move? Retry immediately? Blame the model? Check the p50? All three instincts are wrong. On free tiers, all three.
I keep seeing the same four myths in issue trackers, Discord threads, and code reviews. So here's a myth-busting FAQ with a reproducible probe. The probe is standard-library Python. One file. Any OpenAI-compatible endpoint.
AI promoted everyone to reviewer. Almost nobody reviews the queue in front of the model. That's the gap this post covers.
I build small apps on free model endpoints. When I test harnesses against MonkeyCode's free model access and free server, I watch the same myths appear on day one. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The workflow works on any endpoint, not just theirs.
Myth 1: "The free tier runs a dumber model"
Run the same prompt with temperature 0. Free endpoint or paid endpoint, same text. Compare the outputs. They match token for token.
The weights are the same. Scheduling is different.
Correct mental model: the free tier is a shared queue, not a downgraded brain. Your request waits for an inference slot. Then it runs on the same model as everyone else.
Myth 2: "p50 latency is the health meter"
Queue delay is bimodal. Half your requests land on a warm slot. Half sit behind a cold start or a crowded queue. p50 blends two populations into one number that means nothing.
I've seen p50 look great while one request in five timed out. That's not a healthy service. That's a queue measured wrong.
Correct mental model: watch p95, p99, and the stall rate. Split first-token time from total time. They answer different questions.
Myth 3: "Retry now. Retry hard."
A timeout usually means the queue is crowded at that exact microsecond. Retrying now buys the same ticket to the same line. You're not recovering a request. You're concentrating load.
Worse: five clients retrying in sync create a thundering herd. The queue gets busier. The next timeout becomes more likely.
Correct mental model: retry with jittered exponential backoff. Random sleep that grows each attempt. Spread retries across time instead of stacking them.
Myth 4: "HTTP 200 means the answer is complete"
With streaming, 200 means the gate opened. Nothing more. The first token can still be seconds away. Some servers send headers, then stall.
I probe until data: [DONE]. That's the only honest completion signal. A 200 with no data is a stall. Set a read timeout so the probe can't hang forever.
Correct mental model: first token and last token are separate events. Track both. A small gap means generation. A big gap after the 200 means queue.
The probe
The script checks myths 2, 3, and 4 in one run. Myth 1 needs no script: just diff two responses at temperature 0.
If your endpoint ignores stream, the probe will count every call as a stall. Confirm streaming support first.
#!/usr/bin/env python3
'''myth_probe.py - check free-tier myths on any OpenAI-style endpoint.
Usage:
python myth_probe.py --url https://host/v1/chat/completions \
--model your-model [--key ''] [--burst 20] [--workers 5] [--timeout 30]
Standard library only. Python 3.9+.
'''
import argparse
import json
import random
import time
from concurrent.futures import ThreadPoolExecutor
from urllib.request import Request, urlopen
def pct(vals, p):
if not vals:
return float('nan')
s = sorted(vals)
return s[min(len(s) - 1, len(s) * p // 100)]
def one_call(url, key, model, timeout):
body = {
'model': model,
'stream': True,
'temperature': 0,
'messages': [{'role': 'user',
'content': 'List the numbers 1 to 20, one per line.'}],
}
req = Request(url, data=json.dumps(body).encode(), method='POST', headers={
'Content-Type': 'application/json',
'Authorization': 'Bearer ' + key,
})
t0 = time.monotonic()
first_at = total_at = None
try:
with urlopen(req, timeout=timeout) as r:
saw_first = False
while True:
raw = r.readline()
if not raw:
break
line = raw.decode('utf-8', 'ignore').strip()
if not line.startswith('data:'):
continue
if not saw_first:
first_at = time.monotonic() - t0
saw_first = True
if line == 'data: [DONE]':
total_at = time.monotonic() - t0
break
if first_at is None or total_at is None:
return {'ok': False}
return {'ok': True, 'first': first_at, 'total': total_at}
except Exception:
return {'ok': False}
def burst(url, key, model, workers, n, timeout):
with ThreadPoolExecutor(max_workers=workers) as ex:
futures = [ex.submit(one_call, url, key, model, timeout)
for _ in range(n)]
results = [f.result() for f in futures]
ok = [r for r in results if r['ok']]
print(f'completed {len(ok)}/{n} stall rate {1 - len(ok) / n:.0%}')
for label, field in (('first_token', 'first'), ('total_time', 'total')):
vals = [r[field] for r in ok]
print(f'{label:11s} p50 {pct(vals, 50):5.2f}s '
f'p95 {pct(vals, 95):5.2f}s p99 {pct(vals, 99):5.2f}s')
def client_work(url, key, model, timeout, mode, budget):
sent = 0
for i in range(budget):
sent += 1
if one_call(url, key, model, timeout)['ok']:
return True, sent
if mode == 'backoff':
time.sleep(random.uniform(0, 0.4) * (1.6 ** i))
return False, sent
def herd(url, key, model, timeout, clients, mode):
t0 = time.monotonic()
with ThreadPoolExecutor(max_workers=clients) as ex:
out = list(ex.map(lambda _: client_work(
url, key, model, timeout, mode, 8), range(clients)))
done = sum(1 for ok, _ in out if ok)
sent = sum(s for _, s in out)
print(f'{mode:9s} done {done}/{clients} requests {sent:3d} '
f'wall {time.monotonic() - t0:5.1f}s')
if __name__ == '__main__':
ap = argparse.ArgumentParser()
ap.add_argument('--url', required=True,
help='OpenAI-style /chat/completions URL')
ap.add_argument('--model', required=True)
ap.add_argument('--key', default='')
ap.add_argument('--timeout', type=float, default=30.0)
ap.add_argument('--burst', type=int, default=20)
ap.add_argument('--workers', type=int, default=5)
ap.add_argument('--clients', type=int, default=10)
args = ap.parse_args()
print('== burst ==')
burst(args.url, args.key, args.model, args.workers,
args.burst, args.timeout)
print('== herd (retry storm) ==')
herd(args.url, args.key, args.model, args.timeout,
args.clients, 'immediate')
herd(args.url, args.key, args.model, args.timeout,
args.clients, 'backoff')
How to read the output
Example output — numbers are illustrative. Your queue will look different.
== burst ==
completed 18/20 stall rate 10%
first_token p50 0.82s p95 6.41s p99 11.02s
total_time p50 1.24s p95 7.73s p99 12.90s
== herd (retry storm) ==
immediate done 8/10 requests 47 wall 34.2s
backoff done 10/10 requests 31 wall 28.6s
What each line tells you:
-
completed 18/20— 10% of requests never finished. That's your stall rate. -
first_token p95— six seconds is not a slow model. It's a busy queue. -
total_time p99— eleven seconds to finish twenty tokens. The queue dominates, not the weights. -
immediatevsbackoff— the herd test shows the crowd paying for instant retries. Fewer requests, better completion, faster wall time.
The corrected mental model
- Free tier = shared queue plus the same model, not a downgraded model.
- Timeouts are queue signals, not model verdicts.
- Jittered retries spread load. Immediate retries concentrate it.
- 200 is a gate.
data: [DONE]is the answer. - Measure p95, p99, and stall rate. Ignore p50.
Who should not use this
- Teams with a paid SLA endpoint and stable latency. The probe still works, but the queue is rarely the bottleneck.
- UX-critical flows that need guaranteed latency. Don't engineer around a free tier. Put a paid path behind it. Keep the free tier for best-effort work.
- Anyone who needs one answer and nothing else. Use non-streaming mode with a hard timeout. You just lose the ability to separate queue time from generation time.
Should you abandon free tiers?
No. Free tiers are great for agents, batch jobs, and prototypes. They're wrong where a stall costs real money.
Know which lane you're in before you wire the retry loop. That's the whole FAQ.
The verdict
The free tier is not a toy. It's a queue wearing a model costume. Once you accept that, retries, monitoring, and expectations all fall into place.
Grab any free endpoint. Run the probe. Look at your p99. Then you'll know whether you're fighting a model or a queue. One hour. No dashboards. That's the whole test.
Top comments (3)
This is a useful distinction, especially the separation between queue latency, time-to-first-token, generation time, and completion/stall behavior. A lot of teams collapse all of those into a single “API latency” metric and then draw the wrong conclusions.
One point I’d add is that the “free tier = same model, different queue” claim should be treated as an empirical hypothesis, not a universal rule. Providers can route free and paid traffic through different model revisions, quantization levels, capacity pools, rate limits, or infrastructure. Temperature 0 matching outputs is useful evidence, but it doesn't prove identical weights or execution paths. A stronger test would compare model/version metadata where available and run repeated prompts across sufficiently large samples.
The streaming point is particularly important. I’d instrument at least:
request start → HTTP headers
headers → first meaningful token
first token → final token
final token → [DONE]
timeout / disconnect / malformed-stream rate
That gives you much better observability than p50 alone.
I also like the retry-storm experiment. One further improvement would be to include server-provided Retry-After / rate-limit headers when available and combine them with bounded exponential backoff + full jitter. Otherwise a client can still generate unnecessary traffic even with a well-designed retry policy.
For production systems, I’d probably turn this into a small benchmark harness that continuously records these distributions and compares endpoints over time. The really interesting signal isn't just “which endpoint is faster,” but how latency and failure probability change as concurrency increases.
Great practical write-up. The broader lesson is valuable: measure the actual lifecycle of a request before deciding what component is responsible for the failure.
I work with a small Canada-based remote development team focused on AI, automation, backend systems, and developer tooling. We’re interested in building long-term technical relationships with developers working on projects like this.
Different capacity pools is the case I actually hit. Same model id. Leftover daily capacity on PZERO dies at UTC midnight. Yesterday's row can 503 tonight even if the label is still listed.
Retry now is the myth I dropped. I quote the live row before I queue. Thin book, I shrink the job. A 503 from an empty leftover book is not a timeout. Backoff does not refill it.
That’s a good distinction. If the provider has separate capacity pools, then the model ID alone is clearly not enough to infer equivalent serving conditions. A “same model” response can still behave very differently depending on the pool, quota state, routing policy, and remaining capacity.
I especially like your point about checking the live capacity signal before queueing work. In that case, I’d treat a 503 caused by exhausted capacity as an admission-control signal rather than a transient transport failure. Retrying it blindly just adds pressure without increasing the probability of success.
A robust client could therefore separate failures into categories such as:
capacity/quota exhausted → defer or route to another pool
429/rate limited → honor Retry-After and back off
5xx/transient infrastructure → bounded jittered retry
stream/read timeout → classify based on whether headers or tokens were received
successful response → validate actual stream completion
That classification seems more valuable than having one generic “retry on failure” policy.
The UTC rollover behavior is also a good reason to record capacity state and failure reason alongside latency metrics. Otherwise an overnight change can easily look like random model instability.
This is exactly the kind of production behavior that’s interesting to benchmark rather than assume. If you’re expanding the probe, I’d be interested in comparing notes.