The First Call Is Not Your Latency: A Warm-Up Gate You Can Reproduce

Does your endpoint get faster, or does your script get luckier?

You run a thirty-call loop against a model endpoint, you average the wall clock, and the number looks reasonable. Then a teammate runs the same loop in the morning, gets something twice as slow, and now you have a disagreement you cannot settle with data. Is the endpoint variable, or is your measurement?

I kept running into this whenever I measured anything on a free tier, because the call after a quiet period can carry a cost that later calls do not. So I stopped benchmarking "the endpoint" and started measuring two separate clocks per call, after deliberately discarding a warm-up phase. Below is the harness I would keep.

This is a field note about a measurement harness, not a vendor report. Every number in the code runs against a local mock I wrote, so you can reproduce all of it on your machine before you point the probe at anything real.

The naive loop averaged three different things into one number

My first pass was the obvious one: a for loop, time.perf_counter() around each request, then statistics.mean. What broke was not the arithmetic — it was the assumption that all thirty samples came from the same population.

• Sample 1 paid for whatever happens after an idle period before the first byte arrives.
• Samples 2-30 told me about steady state, which is the only thing I actually wanted.
• Every sample hid any single gap inside the response, because a mean over total duration cannot see a pause in the middle.

So the average was a blend of cold-start cost, steady-state cost, and an invisible stall, weighted by how many warm-up calls I happened to include. No wonder two runs disagreed. Have you ever checked whether your first sample is even from the same distribution as your last?

Three shapes a flat average cannot distinguish

1. Cold start: time before the first byte, which can dominate a short call and vanish by the third one.
2. Mid-stream stall: a long gap between chunks, which can leave the total duration only mildly worse while the experience is much worse.
3. Steady state: the boring, useful region you probably meant to measure.

A single number collapses all three, and then you tune the wrong thing. If the cold start is your problem, adding retries is noise; if the stall is your problem, warming up is noise. You need a probe that can tell you which one you have.

A stand-in endpoint you can run offline

Before touching any hosted endpoint, I built a mock that can reproduce each shape on demand. It is stdlib-only, it streams chunked responses, and it makes the first call slow so the warm-up logic has something to chew on.

# fake_endpoint.py
# usage: python3 fake_endpoint.py 1.20 2 0.90
#        (cold_start_seconds, stall_at_chunk, stall_seconds)
import sys, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

COLD_START = float(sys.argv[1]) if len(sys.argv) > 1 else 1.20
STALL_AT = int(sys.argv[2]) if len(sys.argv) > 2 else 2
STALL_FOR = float(sys.argv[3]) if len(sys.argv) > 3 else 0.90


class Handler(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"
    calls = 0

    def log_message(self, *args):
        pass  # keep the terminal readable

    def do_POST(self):
        Handler.calls += 1
        length = int(self.headers.get("content-length", 0))
        self.rfile.read(length)

        self.send_response(200)
        self.send_header("content-type", "text/event-stream")
        self.send_header("transfer-encoding", "chunked")
        self.end_headers()

        if Handler.calls == 1:
            time.sleep(COLD_START)  # only the first call pays this

        for i in range(8):
            if i == STALL_AT:
                time.sleep(STALL_FOR)
            payload = b'{"delta":"ok"}\n'
            self.wfile.write(b"%x\r\n%s\r\n" % (len(payload), payload))
            self.wfile.flush()
            time.sleep(0.01)

        self.wfile.write(b"0\r\n\r\n")
        self.wfile.flush()


ThreadingHTTPServer(("127.0.0.1", 8787), Handler).serve_forever()

Start it in one terminal and leave it running. The three arguments are not magic numbers; they are the knobs you turn to reproduce each failure shape deliberately, which is the whole point of having a stand-in.

python3 fake_endpoint.py 1.20 2 0.90

Probing with two clocks and a stall threshold

The probe records time-to-first-byte and time-to-last-byte separately, tracks the largest gap between chunks, and keeps the warm-up phase in its own bucket instead of deleting it silently.

# probe.py
import argparse, json, statistics, time, urllib.request


def one_call(url, timeout=30.0):
    req = urllib.request.Request(
        url,
        data=b'{"prompt":"ping"}',
        headers={"content-type": "application/json"},
    )
    t0 = time.perf_counter()
    ttfb = None
    last = None
    gaps = []

    with urllib.request.urlopen(req, timeout=timeout) as resp:
        for _line in resp:
            now = time.perf_counter()
            if ttfb is None:
                ttfb = now - t0
            if last is not None:
                gaps.append(now - last)
            last = now

    if last is None:
        raise RuntimeError("no body bytes received")
    return {"ttfb": ttfb, "ttlb": last - t0, "max_gap": max(gaps) if gaps else 0.0}


def percentile(values, p):
    ordered = sorted(values)
    return ordered[int(p * (len(ordered) - 1))]


def report(label, rows):
    if not rows:
        return
    print(f"{label}: n={len(rows)}")
    for key in ("ttfb", "ttlb", "max_gap"):
        vals = [r[key] for r in rows]
        print(
            f"  {key:8s} p50={statistics.median(vals) * 1000:7.0f} ms"
            f"  p95={percentile(vals, 0.95) * 1000:7.0f} ms"
        )


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--url", default="http://127.0.0.1:8787/v1/chat")
    ap.add_argument("--warmup", type=int, default=2)
    ap.add_argument("--measured", type=int, default=30)
    ap.add_argument("--stall-gap", type=float, default=0.25)
    args = ap.parse_args()

    warm = [one_call(args.url) for _ in range(args.warmup)]
    rows = [one_call(args.url) for _ in range(args.measured)]

    report("warm-up (discarded, still shown)", warm)
    report("measured", rows)

    stalls = sum(1 for r in rows if r["max_gap"] > args.stall_gap)
    print(f"stalls in measured phase: {stalls}/{len(rows)}")
    print("verdict:", "OK" if stalls == 0 else "REVIEW - stalls present")


if __name__ == "__main__":
    main()

Run it against the mock with the cold start and stall both switched on. With those arguments, the warm-up bucket shows a first-byte time north of a second, the measured bucket drops to tens of milliseconds, and the stall counter still fires on every measured call because the mock pauses at chunk 2 regardless of temperature.

That last detail is the lesson I would repeat to anyone: warm-up fixed the cold start and did nothing for the stall, and the two clocks plus the gap threshold are what let me see the difference instead of averaging it away. What would your current benchmark print if the middle of the stream wandered off for a second?

A decision table for reading your own output

What you see	Most likely cause	What to do
Warm-up ttfb far above measured ttfb	Cold start dominates	Discard warm-up, report measured only
Measured p50 fine, p95 several times higher	Queueing or tail noise	Report p95 and sample count together
max_gap above threshold on most calls	Mid-stream stall	Treat as a transport problem, not model quality
ttfb short, ttlb long	Generation length, not startup	Budget by output size, measure both clocks
Almost no variance across 30 identical calls	Cached or deduplicated response	Vary the payload before trusting it

Where a free endpoint fits into this workflow

The nice property of a mock-first harness is that the expensive part of the experiment costs nothing. Once the probe is trustworthy against the mock, you can point --url at a hosted endpoint and re-run the same gate without rewriting your measurement code. MonkeyCode's free model access and free server option, as supplied by the operator, are one way to get a target for that second phase without a billing setup — verify the current terms yourself before you depend on them, since availability and limits are exactly the kind of detail that changes.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you want a target to point the --url flag at for the measured phase, the free model access and free server option are the obvious place to start.

Limitations, and who should skip this

This is a gate, not a benchmark. Thirty calls on one machine tell you whether your measurement is contaminated; they do not tell you a defensible p99, and I would not quote the output as a service-level statement.

• Skip it if your script makes one call per deploy and nobody cares about the difference between 400 ms and 900 ms.
• Skip it if warm-up requests consume a quota you cannot spare, since two discarded calls per run is still a real cost.
• Skip it if you need statistical rigour, because this harness controls bias rather than establishing confidence intervals.
• Do not skip it if your latency numbers have ever disagreed with a colleague's, because that argument is usually a population problem, not an endpoint problem.

What I would carry forward

Keep the warm-up phase visible instead of quietly deleting it, measure first byte and last byte on separate clocks, and gate on the largest inter-chunk gap rather than the mean. None of that is clever, and all of it costs fifteen minutes before you trust a number. The part I would drop is the instinct to report a single average and call it latency — what is the smallest change that would make your current benchmark honest?