I used to trust the first successful request.
Big mistake.
Sound familiar?
A free model endpoint can look perfect at 1 request.
At 10 concurrent requests, it becomes a different product.
Same URL. Same token. Different behavior.
This is a myth-busting FAQ about concurrency, queues, and free model servers.
I point this probe at MonkeyCode's free model access and its free server option.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Myth 1: 'A free endpoint is one server'
It is not.
A free endpoint is a shared scheduler.
Many tenants share the same pool.
Your request waits in a queue.
So does everyone else's.
That explains the weird latency spikes.
They are not random.
They are queueing.
Myth 2: 'If it responds once, it will respond under load'
False.
The first request warms a path.
The 20th request may hit a cold path.
Cold paths are slower.
Sometimes they fail.
I learned this the hard way.
My demo worked.
My load test did not.
The probe: load_probe.py
Here is the script I use.
It sends concurrent requests.
Then it reports status codes and latency percentiles.
# load_probe.py
import asyncio
import sys
import time
import httpx
URL = 'https://your-endpoint.example/v1/chat/completions'
TOKEN = 'your-token'
PROMPT = 'Reply with the word ok.'
async def one(client, sem, i):
async with sem:
start = time.perf_counter()
try:
response = await client.post(
URL,
headers={'Authorization': f'Bearer {TOKEN}'},
json={
'model': 'free-model',
'messages': [{'role': 'user', 'content': PROMPT}],
'max_tokens': 10,
},
timeout=30,
)
elapsed = time.perf_counter() - start
return {'i': i, 'status': response.status_code, 'seconds': round(elapsed, 3)}
except Exception as exc:
elapsed = time.perf_counter() - start
return {'i': i, 'status': 'error', 'seconds': round(elapsed, 3), 'error': type(exc).__name__}
async def run(concurrency, total):
sem = asyncio.Semaphore(concurrency)
async with httpx.AsyncClient() as client:
tasks = [one(client, sem, i) for i in range(total)]
return await asyncio.gather(*tasks)
if __name__ == '__main__':
concurrency = int(sys.argv[1]) if len(sys.argv) > 1 else 5
total = int(sys.argv[2]) if len(sys.argv) > 2 else 25
results = asyncio.run(run(concurrency, total))
statuses = {}
for r in results:
statuses[r['status']] = statuses.get(r['status'], 0) + 1
ok = [r for r in results if r['status'] == 200]
errors = [r for r in results if r.get('status') == 'error']
print('statuses:', statuses)
print('ok:', len(ok), 'errors:', len(errors))
if ok:
latencies = sorted(r['seconds'] for r in ok)
p50 = latencies[len(latencies) // 2]
p95 = latencies[min(len(latencies) - 1, int(len(latencies) * 0.95))]
print('p50:', p50, 'p95:', p95)
Replace the URL and token.
Then run it.
python load_probe.py 1 10
python load_probe.py 5 25
python load_probe.py 10 50
Example output
Here is an illustrative run.
Your numbers will differ.
statuses: {200: 18, 429: 5, 'error': 2}
ok: 18 errors: 2
p50: 2.1 p95: 11.4
Read the output like this.
- 200 count tells you how many requests completed.
- 429 means the shared pool asked you to slow down.
- error means a timeout, reset, or DNS failure.
- The gap between p50 and p95 reveals queueing.
A small gap means stable service.
A large gap means the pool is congested.
Myth 3: 'Retries fix everything'
Retries do not fix congestion.
They amplify it.
Every instant retry adds more load.
The pool gets worse.
Then everyone retries.
That is a stampede.
Use a retry ladder instead.
| Response | What it means | What to do |
|---|---|---|
| 200 | Success | Keep going |
| 429 | Too many requests | Exponential backoff + jitter |
| 5xx | Server-side failure | Retry once after a short delay |
| Timeout | Unknown | Retry only if the request is idempotent |
| Connection error | Pool is saturated | Wait and retry with backoff |
Never retry instantly.
That turns a spike into a stampede.
Myth 4: 'Latency is a single number'
Latency is a distribution.
A single sample tells you almost nothing.
The p50 tells you the typical case.
The p95 tells you the worst case you will feel.
I stopped trusting single samples.
I started trusting percentiles.
Myth 5: 'A bigger concurrency number is better'
More concurrency does not mean more throughput.
It means more queueing.
The pool has a limit.
Your client does not know where it is.
Run the probe with concurrency=1.
Then run it with concurrency=10.
Compare the p95.
If the p95 jumps, you found the queue.
If errors appear, you found the edge.
That edge is the real limit.
The advertised limit is not.
The corrected mental model
Replace the old model with this one.
- A free endpoint is a shared scheduler, not a dedicated box.
- The first request warms a path. Later requests may not.
- Concurrency is a first-class variable, not an afterthought.
- Measure distributions, not single samples.
- Add a bounded queue on your side.
Your client is part of the system.
If you send 50 requests at once, you are the load.
What this probe does not measure
This probe has limits.
It does not measure end-to-end streaming latency.
It does not test multi-turn conversations.
It does not measure model quality.
It only measures one thing: how the endpoint behaves under concurrency.
That is enough to bust the myths.
Who should not use this approach
Do not use free endpoints for production traffic with a strict SLA.
Do not use them for bursty workloads without a buffer.
Do not use them when variable latency is unacceptable.
Use this probe for demos, prototypes, and honest evaluations.
That is the right scope.
A note on changing free tiers
Free tiers change.
They are not contracts.
An endpoint that works today may fail tomorrow.
A quota that exists today may disappear.
Re-run the probe when the endpoint changes.
Re-run it when your traffic pattern changes.
That is the only honest way to stay current.
The point
The myth is not that free endpoints are bad.
The myth is that they behave like paid ones.
They do not.
They behave like shared infrastructure.
Test before you trust.
Then test again.
I keep this script in every project that touches a free endpoint.
You might want one too.
Top comments (0)