Your Timeout Is a Guess: A Latency Probe and Go/No-Go Table for Free-Tier Model Endpoints

Your client timeout is a promise about someone else's server. Most teams never check whether that promise is true — they copy timeout=30, retries=2 from a tutorial, ship it, and discover the real distribution during an incident.

This post is a way to stop guessing. You will measure a free-tier model endpoint yourself, turn the samples into a decision table, and block promotion when the evidence is too thin to support a number. The artifact is a dependency-free probe script plus a checklist you can paste into a pull request.

The number nobody measures

Latency percentiles on a shared endpoint are not a property of the model. They are a property of the model, the queue in front of it, your region, and the hour of the day.

That means an average latency figure from a vendor blog post tells you almost nothing about your own traffic path. You need your own samples, taken from the place your code actually runs, with the prompt sizes you actually send.

And you need the failure side, not just the success side. A probe that only records successful calls will happily report a p95 of 900 ms while 12% of requests were timing out — because the timeouts never made it into the sample set.

What to measure

Four buckets, kept separate. Mixing them produces a number that describes nothing.

• Cold requests. First call after idle. Often 5-20x slower than warm calls, and it is the one your first user of the day experiences.
• Warm requests. Steady state, sequential, one in flight.
• Concurrent requests. Two or four in flight. Free tiers frequently serialize or throttle here, so this bucket reveals the throughput ceiling you actually have.
• Errors by class. 429, 5xx, read timeouts, connection resets, and truncated bodies are five different bugs and deserve five different counters.

For streaming endpoints, also record time to first token separately from total time. A 6-second total with a 400 ms first token feels fast; the reverse feels broken.

The probe

Standard library only, so you can run it on a minimal box. It is a template — I have not run it against your endpoint, and the paths assume an OpenAI-compatible /chat/completions shape. Adjust and verify before you trust any of it.

# probe.py — sequential latency sampler, stdlib only
import json, os, time, urllib.request, urllib.error

BASE  = os.environ["AI_BASE_URL"].rstrip("/")   # e.g. https://host/v1
KEY   = os.environ["AI_API_KEY"]
MODEL = os.environ["AI_MODEL"]

def one_call(prompt, timeout_s=90.0):
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 64,
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/chat/completions", data=body, method="POST",
        headers={"Authorization": f"Bearer {KEY}",
                 "Content-Type": "application/json"},
    )
    t0 = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=timeout_s) as r:
            raw = r.read()
        json.loads(raw)                      # reject truncated payloads
        return {"ok": True, "status": 200, "elapsed": time.perf_counter() - t0}
    except urllib.error.HTTPError as e:
        return {"ok": False, "status": e.code, "elapsed": time.perf_counter() - t0}
    except Exception as e:
        return {"ok": False, "status": type(e).__name__, "elapsed": time.perf_counter() - t0}

def run(n=60, gap_s=1.0, out="probe.jsonl"):
    with open(out, "w") as f:
        for i in range(n):
            rec = one_call("Reply with the single word: ok")
            rec["i"] = i
            f.write(json.dumps(rec) + "\n")
            time.sleep(gap_s)
    return out

if __name__ == "__main__":
    print(run(int(os.environ.get("PROBE_N", "60"))))

Run it once cold (immediately after an idle period) and once warm. Keep both files; they answer different questions.

Then reduce the samples with nearest-rank percentiles, and — this part matters — compute them over all records, counting errors at their measured elapsed time rather than dropping them.

# summarize.py
def pct(values, p):
    if not values:
        return None
    s = sorted(values)
    k = max(0, min(len(s) - 1, int(round(p / 100 * len(s) + 0.5)) - 1))
    return s[k]

import json
recs = [json.loads(l) for l in open("probe.jsonl")]
n     = len(recs)
errors = [r for r in recs if not r["ok"]]
times  = [r["elapsed"] for r in recs]
print({
    "n": n,
    "error_rate": round(len(errors) / n, 3) if n else None,
    "p50": pct(times, 50), "p95": pct(times, 95), "max": max(times or [0]),
})

Turning samples into a decision

Now the part teams skip: the numbers have to change the config, or the measurement was theater.

Use p95 as the timeout floor, not the mean. A timeout equal to the mean guarantees you cut off roughly half your slow-but-fine requests — the worst possible choice, because the request succeeds on the server and you pay for it anyway.

Signal from the probe	Timeout	Retries	Concurrency	Verdict
n < 30 successful samples	unchanged	0	1	Block — evidence too thin
error rate > 5% at sequential load	2 × p95	1, jittered	1	Block — fix errors first
p95 warm < 3s, cold < 10s	2 × p95 warm	1	2	Ship with a slow-path message
cold p95 > 4 × warm p95	2 × cold p95	1	1	Ship, but warm on deploy
p95 at concurrency 4 > 3 × p95 at 1	2 × p95 @1	1	1	Ship single-flight; queue the rest
any truncated body observed	2 × p95	1	1	Block — parse layer is unsafe

Read the table as a starting point, not a law. Your acceptable numbers depend on whether a slow answer is merely annoying or costs money.

The checklist you can paste into a PR

Each line needs an artifact, not an opinion. A checkbox without a file attached is a wish.

1. Evidence exists. probe.jsonl plus probe.summary.json committed, dated, with the endpoint hostname and the region you ran from. No file, no merge.
2. Sample floor met. At least 30 sequential samples per bucket you claim to cover. Below that, the summary says INSUFFICIENT and the checklist fails closed.
3. Cold and warm are separated. One file, one bucket; never averaged together.
4. Errors are counted, not filtered. The summary reports error_rate; a zero-error run with 12 samples is treated as unmeasured.
5. Timeout derives from evidence. A comment next to the config constant explains which percentile produced it and links the summary file.
6. Retries are bounded and jittered. A retry you cannot count is an outage you cannot explain later.
7. The evidence has a date. Anything older than 30 days is stale for a free tier; re-run before promoting.
8. Degradation path is named. If p95 doubles tomorrow, the checklist points at the file that defines the fallback.

One extra rule I would add for any shared workload: measure with the largest prompt your feature actually sends. A 64-token probe understates the latency of a 4,000-token one by a lot.

Where this fits in a free-tier workflow

Free access is exactly where this discipline pays off, because the variance is highest and the consequences of guessing are cheapest to avoid.

MonkeyCode — with free model access and a free server option, per the operator — is convenient for this specific job: the free model access means you can take repeated sample runs without treating each one as a budget decision, and running the probe on the free server gives you a sample set from a stable network location rather than your laptop on hotel Wi-Fi.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

That said, the method is portable. Any endpoint, any key, the probe script above.

Limitations, stated plainly

• Your probe defines your workload, not everyone's. Different prompt lengths, systems messages, and max_tokens will shift the distribution.
• Small samples lie quietly. At n = 15, a p95 is barely a p95. Treat anything under 30 as directional and under 100 as approximate.
• A time window is not a contract. Free tiers can change behavior without notice; a green checklist from last month proves last month.
• Region skew is real. A probe from a different zone than production measures the wrong thing.
• Non-streaming probes miss the felt latency. If your feature streams, measure time to first token too.

Who should skip this

If you have a paid endpoint with a contractual latency SLA and a dashboard that already reports p95 by route, this post is redundant — use the dashboard.

Likewise, if your feature is a background job with no human waiting, timeout tuning matters far less than idempotency and duplicate delivery. Measure the thing your users feel.

For the rest of us running a prototype on someone else's free allowance: sample once, write the number down, and let the number — not the tutorial — set your timeout. If you want a starting environment to run the probe in, the free server option is a reasonable place to point it at.