Set Your Timeouts From Data: A 60-Request Probe Before You Ship on a Free Model Endpoint

Free model access makes the first deploy cheap. It does not make it fast, and that gap is where solo builds break. The bill stays at zero and the risk simply relocates: the first real incident is usually a completion that takes nine seconds inside a UI written for instant feedback. Two numbers prevent it, and both are measurable before the first user signs in — p95 latency and the share of calls rejected with a rate-limit status.

This walkthrough builds a 60-request probe, converts its output into named constants, and locks those constants behind a test. Everything below is stdlib Python except a single test dependency, so the measurement costs nothing but ten minutes.

What the probe measures, and what it refuses to measure

Three questions matter at this stage: how long a tiny completion takes at the 95th percentile, how often the endpoint answers with 429, and whether parallel callers make both numbers worse.

Quality, cost per token, and long-context behaviour are excluded on purpose. Those need a golden set and a separate harness; folding them into a latency probe produces one number that describes nothing.

One rule is baked into the script: it aborts after the second rate-limit response. A 429 is an instruction to stop, and a loop that retries around it turns a ten-minute measurement into a moderation problem.

Step 1 — Freeze the request

Variance in the request becomes variance in the measurement. Pin four fields: a fixed prompt, max_tokens of 4, temperature 0, and one model identifier. The reply is discarded; only the timing is kept.

Record the wall-clock hour and the network path as well. A p95 collected at 03:00 from a laptop on hotel Wi-Fi describes neither the endpoint nor the user.

Step 2 — The probe script

Save this as probe.py:

#!/usr/bin/env python3
"""Minimal latency / rate-limit probe. Stdlib only."""
import argparse
import json
import os
import threading
import time
import urllib.error
import urllib.request
from concurrent.futures import ThreadPoolExecutor

PROMPT = "Reply with the single word: ok"


def one_call(base, key, model, timeout_s):
    payload = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": PROMPT}],
        "max_tokens": 4,
        "temperature": 0,
    }).encode("utf-8")
    req = urllib.request.Request(
        base.rstrip("/") + "/chat/completions",
        data=payload,
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer " + key,
        },
        method="POST",
    )
    started = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=timeout_s) as resp:
            resp.read()
            status = resp.status
    except urllib.error.HTTPError as err:
        status = err.code
    except Exception as err:  # timeouts, DNS, TLS
        status = type(err).__name__
    return {"status": status, "seconds": time.perf_counter() - started}


def quantile(values, q):
    if not values:
        return None
    ordered = sorted(values)
    idx = min(len(ordered) - 1, int(round(q * (len(ordered) - 1))))
    return round(ordered[idx], 3)


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--n", type=int, default=60)
    ap.add_argument("--concurrency", type=int, default=1)
    ap.add_argument("--sleep", type=float, default=1.0)
    ap.add_argument("--timeout", type=float, default=60.0)
    ap.add_argument("--max-429", type=int, default=2)
    ap.add_argument("--out", default="probe.json")
    args = ap.parse_args()

    base = os.environ["MC_API_BASE"]
    key = os.environ["MC_API_KEY"]
    model = os.environ["MC_MODEL"]

    stop = threading.Event()
    lock = threading.Lock()
    results = []
    limited = 0

    def worker(_):
        nonlocal limited
        if stop.is_set():
            return
        r = one_call(base, key, model, args.timeout)
        with lock:
            results.append(r)
            if r["status"] == 429:
                limited += 1
                if limited >= args.max_429:
                    stop.set()
        time.sleep(args.sleep)

    with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
        list(pool.map(worker, range(args.n)))

    ok = [r["seconds"] for r in results if r["status"] == 200]
    summary = {
        "requests": len(results),
        "ok": len(ok),
        "rate_limited": limited,
        "aborted_on_429": stop.is_set(),
        "p50_s": quantile(ok, 0.50),
        "p95_s": quantile(ok, 0.95),
        "max_s": round(max(ok), 3) if ok else None,
        "concurrency": args.concurrency,
    }
    print(json.dumps(summary, indent=2))
    with open(args.out, "w", encoding="utf-8") as fh:
        json.dump({"summary": summary, "results": results}, fh, indent=2)


if __name__ == "__main__":
    main()

Two runs are enough for a first cut. The first measures a single caller; the second shows whether three parallel callers degrade each other.

export MC_API_BASE="https://<endpoint-from-docs>"
export MC_API_KEY="<your-key>"
export MC_MODEL="<model-id>"

python probe.py --n 60 --concurrency 1 --sleep 1.0 --out probe-c1.json
python probe.py --n 20 --concurrency 3 --sleep 1.0 --out probe-c3.json

A typical output block looks like this fixture — labelled as a fixture, because the real numbers belong to a real run:

{
  "requests": 60,
  "ok": 58,
  "rate_limited": 0,
  "aborted_on_429": false,
  "p50_s": 1.4,
  "p95_s": 3.1,
  "max_s": 6.8,
  "concurrency": 1
}

The gap between p95 and max is itself information. A wide spread on a single caller means queue depth in the client will not help; the endpoint is doing the waiting, so the UI has to say so.

Step 3 — Run the probe where it costs nothing

A probe aimed at a metered endpoint is a probe with a meter attached. MonkeyCode's free model access and free server option is the zero-cost place to run step 2: the script executes on the free server, close to the endpoint, which also strips local network noise out of the p95.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two checks belong before the output is trusted. First, confirm the current request shape in the documentation — the script assumes the OpenAI-compatible /chat/completions shape, and if that assumption is wrong, only one_call() changes. Second, read the current allowance and limits on the site rather than in a screenshot; availability terms are exactly the sort of detail that moves, and the operator describes the offer only as free model access plus a free server option.

Step 4 — Turn the numbers into constants

Keep the mapping in one importable file, latency_contract.py, so the UI and the client library cannot drift apart:

"""Constants derived from a probe run. Change them with data, not with mood."""

PROFILES = {
    "inline":  {"timeout_s": 8,  "concurrency": 1, "ui": "sync response"},
    "spinner": {"timeout_s": 15, "concurrency": 2, "ui": "async with skeleton"},
    "queue":   {"timeout_s": 30, "concurrency": 1, "ui": "job row + poll"},
    "batch":   {"timeout_s": 45, "concurrency": 1, "ui": "off-hours job"},
}


def choose_profile(p95_s, rate_limited_ratio):
    if rate_limited_ratio > 0.05:
        return "batch"
    if p95_s is None:
        return "queue"          # no data yet: assume the slow case
    if p95_s < 2.0:
        return "inline"
    if p95_s < 5.0:
        return "spinner"
    if p95_s < 12.0:
        return "queue"
    return "batch"

The decision table it encodes:

p95, one caller	429 share	Profile	Client timeout	Product behaviour
under 2 s	5% or less	inline	8 s	direct response
2–5 s	5% or less	spinner	15 s	async plus skeleton
5–12 s	5% or less	queue	30 s	job row, poll every 2 s
over 12 s	any	batch	45 s	off-hours job
any	over 5%	batch	45 s	off-hours job

The rate-limit row outranks latency on purpose. A fast endpoint that refuses eight calls in a hundred is a batch endpoint wearing an inline costume.

Step 5 — Pin the choice with four tests

Four assertions are enough to stop a later refactor from quietly reintroducing optimistic timeouts:

# test_latency_contract.py
from latency_contract import PROFILES, choose_profile


def test_fast_endpoint_gets_inline():
    assert choose_profile(1.2, 0.0) == "inline"


def test_rate_limits_override_latency():
    assert choose_profile(1.2, 0.08) == "batch"


def test_unknown_latency_is_not_optimistic():
    assert choose_profile(None, 0.0) == "queue"


def test_timeouts_are_monotonic():
    order = ["inline", "spinner", "queue", "batch"]
    timeouts = [PROFILES[name]["timeout_s"] for name in order]
    assert timeouts == sorted(timeouts)

python -m pytest -q test_latency_contract.py

Step 6 — Let the limit shape one product decision

The temptation is to hide a slow endpoint behind a spinner and hope. The cheaper move is to change one screen: a feature whose p95 lands in the queue profile writes a row into SQLite, returns a job id immediately, and lets the client poll. That is roughly forty lines and no new infrastructure — and it makes the free tier's limits a design input instead of a surprise.

Accept the rest. If the probe says batch, ship the feature as an off-hours job and tell the user when it runs. Shipping today at zero cost is worth more than shipping a fake instant experience that fails at the worst moment.

Limitations and who should skip this

The probe measures one region at one hour. Sixty calls catch an order-of-magnitude problem; they do not produce a number worth publishing, and a shared free tier will move underneath the result.

Skip the whole approach when nobody is waiting on the output — an overnight pipeline does not need a p95 discussion. Skip it as well when latency is a contractual requirement with a p99 target and support escalation, because that is a purchase, not a measurement.

If the numbers matter for a launch happening today, run the two commands above and let them pick the profile. The free model access and free server option keep the experiment at zero, which is the only budget an indie build should spend on finding out.