I Blamed the Model. The Wrapper Was Sleeping.

A slow agent is usually a slow wrapper.

I stopped trusting the model's own latency number. Here is the trap. Your client reports one thing: completion time. That is the span from request sent to final byte read.

Everything in between disappears. Queueing, routing, TLS, cold start, retry sleeps. All of it folds into a single friendly number.

The model gets blamed for a four-second wait. Who measured the parts? Almost nobody. I know because I didn't.

So I wrote a probe. One call, four numbers.

# probe.py — runnable. Numbers below are yours to produce, not mine.
import json, time, urllib.request

ENDPOINT = "http://127.0.0.1:8080/v1/chat/completions"   # your gateway, not mine
BODY = json.dumps({
    "model": "your-model",
    "stream": True,
    "messages": [{"role": "user", "content": "count to five"}],
}).encode()

def one_call():
    req = urllib.request.Request(
        ENDPOINT, data=BODY,
        headers={"Content-Type": "application/json"},
    )
    t0 = time.perf_counter()
    first_byte = None
    chunks = 0
    with urllib.request.urlopen(req, timeout=60) as r:
        for _ in r:                      # each yielded line is one stream chunk
            if first_byte is None:
                first_byte = time.perf_counter()
            chunks += 1
    t1 = time.perf_counter()
    return {
        "ttfb_ms":  round((first_byte - t0) * 1000, 1),
        "drain_ms": round((t1 - first_byte) * 1000, 1),
        "total_ms": round((t1 - t0) * 1000, 1),
        "chunks":   chunks,
    }

runs = [one_call() for _ in range(20)]   # 1 cold call, 19 warm, same process

Four numbers, four suspects. ttfb_ms is queueing, routing, TLS, and cold start. drain_ms is decode and transport. total_ms is what your user feels. chunks tells you whether streaming even works.

Now the part everyone forgets. Wrap the loop. Count the sleeps.

sleeps = []

def with_retries(fn, tries=3, base=0.5):
    for attempt in range(tries):
        try:
            return fn()
        except Exception:
            delay = base * (2 ** attempt)
            sleeps.append(delay)
            time.sleep(delay)
    raise RuntimeError("gave up")

slept_ms = round(sum(sleeps) * 1000, 1)   # invisible to the model's dashboard

Half a second plus one second is 1.5 seconds of pure, silent sleep on the second failure. No trace shows it. No dashboard charts it. The user just waits.

But wait. Isn't exponential backoff correct? Yes, for correctness. No, for perceived latency. Correctness and speed fight here, and most clients pick correctness silently.

So read ratios, not milliseconds. Absolute numbers are noise on shared machines. Ratios survive.

Shape of the run	Likely bottleneck	First move
ttfb over 60% of total, call 1 much slower	cold start or routing	keep the connection warm, run the probe in-region
drain over 70% of total, ttfb flat	decode throughput	smaller model, or an endpoint nearer to you
total spikes on a repeating cadence	retry sleeps	cap tries, add jitter, always log slept_ms
p95 / p50 above 3, same prompt, same box	noisy neighbor	stop benchmarking on shared infrastructure

That last row is the honest one for free tiers. A shared box is still a shared box. If you rented it for zero, someone else rented it for zero too.

What do you need to run this loop? A budget. Twenty calls times five prompt shapes is a hundred calls per iteration. Then you tweak jitter, rerun, and pay again. Measuring your own wrapper gets expensive fast.

That is where the free tier earns its place in this method. I use MonkeyCode's free model access and its free server option as the sandbox for the probe above. The operator advertises a free token allowance (10M at the time of writing) and a free server option; those are operator-supplied claims, program terms change, and you should check the current page before you design around them. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why it fits the workflow: I need many cheap calls to see variance. Variance is the signal. A single fast call proves nothing, and a single slow call proves less. I am not using it to chase a vendor crown, and neither should you.

The loop is small. Measure, patch, measure again.

python probe.py > run-a.json
# change one thing: keep-alive, jitter, retry cap
python probe.py > run-b.json
# diff the four keys, not the wall clock you remember

Now the limits, because there are several.

Twenty calls is a smoke test, not a benchmark. Cold start hides in call one, so keep it and label it. Warmup hides in calls two and three, so keep those too and compare medians of the rest.

A free shared server tells you shape, never absolute milliseconds. Never publish a vendor comparison built on it. That comparison would be dishonest, and it would be about the neighbor, not the vendor.

Chunk boundaries differ across SDKs. Parse the stream with the same client you ship, or chunks will mislead you.

Network path dominates everything else. Same region, same machine, or admit you measured your ISP.

Who should not use this approach? Teams already running distributed tracing, because you have this data and more. Anyone whose contract is the provider's own SLO metrics, because your probe does not override a contract. And batch jobs where nobody watches the first byte, because ttfb is irrelevant there. Optimize what people feel.

One more question worth asking. If your retry loop sleeps 1.5 seconds, did you add that to the model's latency budget? Probably not. Neither did I, until the graph showed a gap nobody could name.

Run the probe once before you swap models. Then ask which of the four numbers moved. If you want a zero-budget place to run it, the free tier is right there.