You set your timeout to 30 seconds. Why 30?
Because it felt safe. I did the same thing once. Then my pipeline started failing in weird ways.
Some requests died at 31 seconds. Others finished at 29. My timeout was a coin flip.
So I ran an experiment. I measured 400 calls to a free model server. Then I derived my timeout from the data.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. The server under test was MonkeyCode's free server option.
Why your timeout is a guess
Most timeout values come from vibes. Someone picked 30. Someone else picked 60. Nobody measured anything.
That matters because free model servers have long tails. The median looks great. The p99 is brutal.
A short timeout kills good requests. A long timeout hangs your pipeline. Both waste time and money.
The fix is simple. Measure the latency distribution. Pick a percentile. Derive the timeout.
The experiment
I wanted three numbers:
- Time to first byte (TTFB)
- Total request duration
- Error rate
I fixed everything else. Same prompt. Same max_tokens. Same concurrency.
The prompt was short. max_tokens was 200. I kept both fixed across all 400 calls.
I fired 400 requests at 8 concurrent workers. Then I waited.
Here's the probe script. It works with any OpenAI-compatible endpoint:
"""Timeout budget probe for OpenAI-compatible endpoints.
Usage:
python3 timeout_probe.py --base-url https://endpoint/v1 \
--api-key $API_KEY --model your-model \
--requests 400 --concurrency 8 --max-tokens 200
"""
import argparse
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
import requests
PROMPT = (
"Explain what a database index is, in plain terms. "
"Cover how it speeds up lookups, the storage cost, "
"and one concrete example with a users table. "
) * 4
def probe(base_url, api_key, model, max_tokens):
started = time.monotonic()
ttfb = None
status = None
error = None
try:
with requests.post(
f"{base_url}/chat/completions",
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
},
json={
"model": model,
"messages": [{"role": "user", "content": PROMPT}],
"max_tokens": max_tokens,
"stream": True,
},
stream=True,
timeout=(30, 180),
) as resp:
status = resp.status_code
for line in resp.iter_lines():
if line:
ttfb = time.monotonic() - started
break
except requests.exceptions.Timeout:
error = "timeout"
except requests.exceptions.ConnectionError:
error = "connection_error"
except Exception as exc:
error = type(exc).__name__
return {
"status": status,
"ttfb": ttfb,
"total": time.monotonic() - started,
"error": error,
}
def percentile(values, p):
ordered = sorted(v for v in values if v is not None)
if not ordered:
return float("nan")
k = (len(ordered) - 1) * (p / 100.0)
f = int(k)
c = min(f + 1, len(ordered) - 1)
return ordered[f] + (ordered[c] - ordered[f]) * (k - f)
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--base-url", required=True)
parser.add_argument("--api-key", required=True)
parser.add_argument("--model", required=True)
parser.add_argument("--requests", type=int, default=400)
parser.add_argument("--concurrency", type=int, default=8)
parser.add_argument("--max-tokens", type=int, default=200)
args = parser.parse_args()
results = []
with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
futures = [
pool.submit(
probe, args.base_url, args.api_key, args.model, args.max_tokens
)
for _ in range(args.requests)
]
for future in as_completed(futures):
results.append(future.result())
ok = [r for r in results if r["error"] is None and r["status"] == 200]
ttfb = [r["ttfb"] for r in ok]
total = [r["total"] for r in ok]
print(f"total requests : {len(results)}")
print(f"successes : {len(ok)}")
print(f"errors : {len(results) - len(ok)}")
print()
print(f"ttfb median : {percentile(ttfb, 50):.2f}s")
print(f"ttfb p90 : {percentile(ttfb, 90):.2f}s")
print(f"ttfb p95 : {percentile(ttfb, 95):.2f}s")
print(f"ttfb p99 : {percentile(ttfb, 99):.2f}s")
print()
print(f"total median : {percentile(total, 50):.2f}s")
print(f"total p90 : {percentile(total, 90):.2f}s")
print(f"total p95 : {percentile(total, 95):.2f}s")
print(f"total p99 : {percentile(total, 99):.2f}s")
if __name__ == "__main__":
main()
What the data said
Here's what I got after 400 requests:
| Metric | Median | p90 | p95 | p99 |
|---|---|---|---|---|
| TTFB | 1.8s | 4.5s | 8.0s | 30.1s |
| Total duration | 9.5s | 17.2s | 26.4s | 58.3s |
The error breakdown:
- 372 successes (93%)
- 12 timeouts (client gave up)
- 9 connection resets
- 5 HTTP 429 responses
- 2 malformed streams
The server performs well in one place. The middle. Typical requests finish in under 10 seconds.
It breaks in two places. The tail latency spikes. And errors cluster in bursts.
The median looks great. 1.8s to first byte. 9.5s total.
The tail is the story. p99 total is 6x the median.
Deriving the timeout
Now the math. I used one rule:
timeout = ceil(percentile * margin)
Three profiles:
| Profile | Base | Margin | Timeout |
|---|---|---|---|
| Aggressive | p90 = 17.2s | 1.2x | 21s |
| Balanced | p95 = 26.4s | 1.5x | 40s |
| Safe | p99 = 58.3s | 1.3x | 76s |
Why the margin? Percentiles are estimates. The next 100 requests will differ. The margin absorbs sampling noise.
I picked the balanced profile. 40 seconds. Then I validated it.
The validation run
I ran 100 more requests. This time I enforced a 40s client timeout.
- 94 finished inside the budget
- 4 hit the 40s ceiling
- 2 failed for other reasons
The budget caught 95.9% of requests that would have succeeded. It also capped my worst-case wait at 40s. Without it, I waited up to 180s.
Then I tested the aggressive profile. 21s caught 88.8%. It also killed 11 requests that would have succeeded.
The safe profile caught everything. But a hung request blocked a worker for 76 seconds.
Here's how I apply the budget in real code:
import httpx
import logging
TIMEOUT_BUDGET = 40.0 # derived from p95 * 1.5
client = httpx.Client(timeout=TIMEOUT_BUDGET)
try:
resp = client.post(
f"{BASE_URL}/chat/completions",
headers={"Authorization": f"Bearer {API_KEY}"},
json={
"model": MODEL,
"messages": [{"role": "user", "content": prompt}],
"stream": True,
},
)
except httpx.TimeoutException:
logging.warning("request exceeded %ss budget", TIMEOUT_BUDGET)
# The budget already accounts for the tail.
# Record it and move on. Don't retry blindly.
Why not just use p99?
Good question. p99 sounds safer. But p99 is noisy.
With 400 samples, p99 is driven by four requests. Those four could be a temporary spike. Or a network blip.
A 76s timeout also means a long worst-case hang. Every stuck request blocks a worker for over a minute.
The balanced profile trades a little catch rate for a much shorter hang. That's the right deal for interactive pipelines.
Where this method breaks
Be honest about the limits.
- Sample size. 400 requests is a start, not a guarantee. Run it at different times of day.
- Prompt length. My prompt was short. Long prompts shift the whole distribution.
- Concurrency. I used 8 workers. Your load profile may differ.
- Server changes. Free servers get busy. Models get swapped. Your budget expires.
- Streaming vs non-streaming. I measured streaming. Non-streaming TTFB is different.
Re-run the probe monthly. Or after any major change.
Who should not use this
- Batch jobs with no user waiting. Nobody is waiting. Use a generous budget and retry with backoff.
- Single-shot scripts. One call a day? Just use 120s. The measurement cost isn't worth it.
- Real-time chat UIs. A 40s timeout is terrible UX. You need streaming plus a UI-level deadline.
The takeaway
Your timeout should be a measurement. Not a guess.
Run the probe. Read the percentiles. Pick a profile. Validate it.
The free server I tested was fast in the middle. Slow at the edges. Your endpoint will have its own shape.
Now you know how to find it.
If you want a free endpoint to practice on, MonkeyCode's free server option is where I ran this probe. The script above needs no changes.
Top comments (0)