Every free endpoint has a ceiling. The docs never mention it. The first fifty calls never reveal it.
Then a batch job hits it. Everything slows down. Or fails. Or silently returns garbage.
I wanted one number. At what concurrency does this server stop being useful? So I built a staircase test. This post is the test, the script, and the three failure signatures.
Why a staircase?
A flat load test hides the cliff. Ten parallel calls look fine. Twenty look fine. Then forty collapse. You need to walk toward the edge. One step at a time.
The staircase does exactly that. Start with one concurrent request. Double it each round. Watch the percentiles. Stop when errors cross your threshold.
The three ways a free server breaks
Free endpoints fail in three patterns. Learn to recognize all three.
1. The latency cliff
The p50 stays flat. The p95 triples. The server queues requests instead of rejecting them. Your batch job slows down. Then it stalls.
2. The error spike
429s and 5xxs appear at the same concurrency. The server is telling you the limit. Most clients ignore the message. Then retry logic makes it worse.
3. Silent corruption
HTTP 200. Complete response. Truncated JSON. Or an empty completion. Your pipeline parses it. Your logs don't complain. You ship garbage.
The third one is the worst. It doesn't look like a failure. It looks like success.
The experiment design
Keep the variables boring. Change only one thing: concurrency.
- Fixed prompt. Fixed
max_tokens. - Staircase: 1, 2, 4, 8, 16 concurrent requests.
- Ten requests per step.
- Record p50, p95, p99, success rate, tokens/sec.
- Stop early when errors cross 10%.
Why ten requests? Enough to see a pattern. Few enough to finish fast. This is a ceiling test. Not a benchmark suite.
The script
Save this as ceiling_test.py. It needs Python 3.9+ and httpx.
"""ceiling_test.py — find where a free model endpoint breaks."""
import asyncio
import os
import time
import httpx
ENDPOINT = os.environ.get("LLM_ENDPOINT", "http://localhost:8000/v1/chat/completions")
API_KEY = os.environ.get("LLM_API_KEY", "")
MODEL = os.environ.get("LLM_MODEL", "default")
PROMPT = "Write a haiku about a queue that never drains."
STEPS = [1, 2, 4, 8, 16]
REQUESTS_PER_STEP = 10
TIMEOUT = 30.0
async def one_call(client, sem):
async with sem:
t0 = time.perf_counter()
try:
r = await client.post(
ENDPOINT,
json={
"model": MODEL,
"messages": [{"role": "user", "content": PROMPT}],
"max_tokens": 128,
},
timeout=TIMEOUT,
)
dt = time.perf_counter() - t0
body = r.json()
tokens = body.get("usage", {}).get("total_tokens", 0)
return {"ok": r.status_code == 200, "status": r.status_code,
"latency": dt, "tokens": tokens, "error": ""}
except Exception as exc:
dt = time.perf_counter() - t0
return {"ok": False, "status": 0, "latency": dt,
"tokens": 0, "error": type(exc).__name__}
async def run_step(client, concurrency, count):
sem = asyncio.Semaphore(concurrency)
results = await asyncio.gather(
*[one_call(client, sem) for _ in range(count)]
)
lats = sorted(r["latency"] for r in results)
ok = [r for r in results if r["ok"]]
def pct(p):
idx = min(len(lats) - 1, int(len(lats) * p))
return lats[idx]
errors = sorted({r["error"] or str(r["status"])
for r in results if not r["ok"]})
total_latency = sum(r["latency"] for r in ok)
tokens_per_sec = (
sum(r["tokens"] for r in ok) / total_latency
if total_latency > 0 else 0.0
)
return {
"concurrency": concurrency,
"success": f"{len(ok)}/{len(results)}",
"p50": round(pct(0.50), 2),
"p95": round(pct(0.95), 2),
"p99": round(pct(0.99), 2),
"tokens/sec": round(tokens_per_sec, 1),
"errors": errors,
}
async def main():
headers = {"Authorization": f"Bearer {API_KEY}"} if API_KEY else {}
async with httpx.AsyncClient(headers=headers) as client:
print(f"{'conc':>4} {'success':>8} {'p50':>6} {'p95':>6} "
f"{'p99':>6} {'tok/s':>7} errors")
for step in STEPS:
row = await run_step(client, step, REQUESTS_PER_STEP)
errs = ",".join(row["errors"]) or "-"
print(f"{row['concurrency']:>4} {row['success']:>8} "
f"{row['p50']:>6} {row['p95']:>6} {row['p99']:>6} "
f"{row['tokens/sec']:>7} {errs}")
if __name__ == "__main__":
asyncio.run(main())
Run it:
export LLM_ENDPOINT="https://your-endpoint.example/v1/chat/completions"
export LLM_API_KEY="your-key"
export LLM_MODEL="your-model"
python ceiling_test.py
Reading the output
Here is a sample run. Your numbers will differ. That is the point.
conc success p50 p95 p99 tok/s errors
1 10/10 0.82 1.10 1.24 45.2 -
2 10/10 0.90 1.31 1.55 43.8 -
4 10/10 1.21 2.40 3.10 39.1 -
8 9/10 2.10 8.40 12.20 22.4 529
16 6/10 3.50 25.10 28.00 9.8 529,timeout
Read it left to right. Success rate first. Then the percentiles. Then the errors.
In this sample, the ceiling sits between concurrency 4 and 8. The p95 jumps from 2.4 to 8.4 seconds. Errors appear. Throughput collapses.
That is the number you need. Not the p50. The point where the server stops being useful.
The decision table
| Signature | What it means | What I do |
|---|---|---|
| p95 jumps 3x, p50 flat | Server queues requests | Stay below that concurrency |
| 429s at one step | Rate limit reached | Add backoff, spread the batch |
| 200 with truncated body | Silent corruption | Validate JSON before parsing |
| Timeouts at one step | Hard ceiling | Split the job into smaller chunks |
Keep this table next to the script. It turns raw numbers into decisions.
What the test taught me
Three lessons. Each one cost me a late night.
The mean hides the cliff
Average latency looked fine. The p95 told the truth. Percentiles are the only honest summary.
Free servers degrade gracefully. Until they don't.
The curve looks smooth. Then it falls off a table. There is no warning slope.
The ceiling moves
Prompt length changes it. Time of day changes it. Server load changes it. Re-run before big batches. Not after.
Where I use this with MonkeyCode
When I need a free server for a side project, I point this harness at it first. MonkeyCode offers free model access and a free server option. I treat those like any other endpoint: measure before you trust. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The script takes fifteen minutes. It saves me from scheduling a batch job against a server that dies at step eight.
Who should NOT use this approach
This test is not for everyone. Skip it if:
- You make a few interactive calls per day.
- You use one-off chat, not batch workloads.
- You need a formal SLA. Free servers have none.
- You cannot tolerate a single dropped request. Pay for a server.
The ceiling test answers one question: where does it break? It does not fix the break. It does not promise uptime. It gives you a number. Then you decide.
The closing thought
The ceiling is the spec nobody ships. The docs describe the happy path. The happy path ends at concurrency one.
Run the script. Save the output. The next time an endpoint fails, you will already know why.
Top comments (0)