Jordan Huang

Posted on Aug 21

Your Timeout Is a Guess. Here's the 20-Minute Calibration for Free Model Servers.

#devops #ai #python #performance

Thirty seconds. That was my timeout. Why? It felt safe. It was not.

I watched a batch job die twice. First, requests failed before the server warmed up. Then, one hung request blocked everything behind it. My timeout was the problem. Not the model. Not the network. My guess.

A timeout is a latency budget. Guess it, and you pay twice. Too short means false failures. Too long means silent stalls. Retries only amplify both. I learned that the hard way in my last experiment.

So I stopped guessing. I measured. Here is the 20-minute calibration I now run against any free model server.

The Failure Mode Nobody Measures

Most timeout configs come from vibes. Someone set 30 seconds in 2024. Nobody revisited it. The server changed. The model changed. The workload changed. The timeout did not.

That mismatch is invisible. Your logs show a timeout error. You blame the server. Sometimes you are right. Often you are wrong.

The fix is boring. Measure the real distribution of time-to-first-token (TTFT). Then set a timeout that respects it.

The Experiment Design

Keep it small. Keep it reproducible.

One endpoint.
One fixed prompt.
100 sequential requests.
Record TTFT and total duration.
Compute percentiles.

Why TTFT? Because that is what your timeout guards. If the first token never arrives, the request is dead. Everything after that is streaming.

Why 100? Enough to see the tail. Not enough to annoy the server. If you are polite, the server stays stable.

The Probe Script

I use Python and httpx. Streaming mode matters. A non-streaming request hides the TTFT.

import asyncio
import json
import time
import httpx

URL = "https://your-endpoint.example/v1/chat/completions"
PROMPT = "Write a short paragraph about distributed systems. Keep it under 80 words."
N = 100

async def probe(client, i):
    payload = {
        "model": "your-model",
        "messages": [{"role": "user", "content": PROMPT}],
        "stream": True,
    }
    t0 = time.perf_counter()
    ttft = None
    async with client.stream("POST", URL, json=payload) as resp:
        async for line in resp.aiter_lines():
            if line.startswith("data:") and ttft is None:
                ttft = time.perf_counter() - t0
    total = time.perf_counter() - t0
    return {"i": i, "ttft": ttft, "total": total}

async def main():
    async with httpx.AsyncClient(timeout=None) as client:
        results = await asyncio.gather(*[probe(client, i) for i in range(N)])
    print(json.dumps(results, indent=2))

asyncio.run(main())

Save it as probe.py. Run it. Wait. Drink coffee.

The Percentile Calculator

Raw numbers are noise. Percentiles are signal.

def pct(values, p):
    values = sorted(values)
    k = (len(values) - 1) * p
    f = int(k)
    c = min(f + 1, len(values) - 1)
    return values[f] + (values[c] - values[f]) * (k - f)

ttfts = [r["ttft"] for r in results if r["ttft"] is not None]
totals = [r["total"] for r in results]

print(f"p50  ttft={pct(ttfts, 0.50):.2f}s  total={pct(totals, 0.50):.2f}s")
print(f"p90  ttft={pct(ttfts, 0.90):.2f}s  total={pct(totals, 0.90):.2f}s")
print(f"p95  ttft={pct(ttfts, 0.95):.2f}s  total={pct(totals, 0.95):.2f}s")
print(f"p99  ttft={pct(ttfts, 0.99):.2f}s  total={pct(totals, 0.99):.2f}s")
print(f"max  ttft={max(ttfts):.2f}s  total={max(totals):.2f}s")

What the Output Looks Like

Here is a sample from one of my runs. Treat it as illustrative. Your server will differ. That is the point.

percentile   ttft (s)   total (s)
p50          1.2        4.8
p90          3.4        9.1
p95          5.1        12.6
p99          9.8        21.3
max          22.4       41.7

Look at that gap. p50 to p99 is an 8x jump. A timeout tuned for the median would kill 1% of requests. A timeout tuned for the max would stall every batch.

The Calibration Rule

Now the math is simple.

p50 tells you the happy path.
p95 tells you the normal worst case.
p99 tells you the rare spike.
Set timeout = p99 × 1.5, rounded up.

Here is the decision table I use:

If p99 TTFT is	Set timeout to
under 3s	5s
3–6s	10s
6–12s	20s
over 12s	investigate first

If p99 is over 12 seconds, do not tune. Investigate. The server may be overloaded. Your prompt may be too long. Your network may be the bottleneck.

Where MonkeyCode Fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I ran this calibration against MonkeyCode's free server option. Free tiers are shared resources. Variance is part of the deal. That makes calibration more important, not less. The method works on any endpoint. MonkeyCode's free option is just a convenient place to practice it.

What This Does Not Tell You

This test has limits. Be honest about them.

One run is one snapshot. Servers change by the hour.
TTFT is not the full story. Streaming stalls happen after the first token.
Sequential requests hide concurrency effects. Parallel load changes everything.
Free servers get swapped under you. Re-run the probe weekly.

Timeouts are also not retries. A good timeout prevents false failures. A bad retry strategy creates new ones. I covered that in my retry experiment.

Who Should Skip This

Not everyone needs calibration.

One request per hour? Skip it. Your timeout barely matters.
Paid SLA with published latencies? Use their numbers.
Batch-only workloads? Measure total duration, not TTFT.
Can't run 100 requests? Run 30. The tail will be noisier. The median still helps.

The 20-Minute Habit

Calibration is not a one-time task. It is a habit.

Run the probe when you onboard a new endpoint. Re-run it when behavior changes. Log the percentile table. Compare it next month.

My timeout is no longer a guess. It is a measured decision. Yours can be too.

If you run the script, post your percentile table in the comments. I am curious where your server lands.

DEV Community