The Shape of Latency: What a Free Model Server's Response Times Reveal

Average latency is a lie. The shape behind it is the truth. I stopped trusting the mean and started reading distributions.

A free model server looks fast. Then one request takes ten seconds. The average still looks fine. The distribution tells a different story.

Why the mean hides the queue

Mean latency hides outliers. Outliers are the queue. A healthy server has a tight spread. A congested server grows a long tail.

You need the tail. The tail predicts your timeouts.

The experiment

I sent 200 identical requests to a free model server. One sentence prompt. No streaming. Just total response time.

Here is the collector script. It uses only the standard library.

import json, time, urllib.request, math, statistics

URL = "https://your-free-model-endpoint/v1/chat"
PROMPT = "Say hello in one sentence."
N = 200

def measure_once():
    payload = json.dumps({"prompt": PROMPT}).encode()
    req = urllib.request.Request(URL, data=payload, headers={"Content-Type": "application/json"})
    start = time.perf_counter()
    with urllib.request.urlopen(req, timeout=60) as resp:
        resp.read()
    return time.perf_counter() - start

times = sorted(measure_once() for _ in range(N))
mean = statistics.mean(times)
p50 = times[N // 2]
p95 = times[int(N * 0.95) - 1]

lam = 1 / mean
theoretical_p95 = -math.log(0.05) / lam

print(f"mean={mean:.3f}s p50={p50:.3f}s p95={p95:.3f}s")
print(f"exponential p95={theoretical_p95:.3f}s")
print(f"ratio={p95/theoretical_p95:.2f}")

Run it. It prints the mean, the percentiles, and a ratio.

Send requests sequentially

Do not parallelize the collection. Parallel requests create your own queue. That pollutes the signal. You want the server's queue, not yours.

One request at a time. Slow but clean.

Streaming changes the measurement

If your endpoint streams, record two times. First token and total duration. The first token reveals thinking time. The total reveals queueing plus generation.

For this test, total time is enough. The queue shows up in both.

Reading the ratio

Fit an exponential distribution to your data. The exponential is the shape of a memoryless queue. If your server has no queue, your response times should follow it.

The theoretical p95 is -ln(0.05) / λ. λ is 1 divided by the mean. Compare that to your actual p95.

Ratio (actual p95 / exponential p95)	What it means
0.8 – 1.2	Healthy, no queue
1.2 – 2.0	Mild queueing, occasional waits
> 2.0	Heavy queueing or multi-tenant noise

Ratio near 1 means the server is idle. Ratio above 2 means your requests are waiting behind others.

A concrete workflow

1. Run the script three times. Use the median ratio.
2. Log the ratio with a timestamp.
3. Set your timeout to the observed p95 plus 50%.
4. Re-run weekly. Track the trend.

A rising ratio is an early warning. A falling ratio means the server is healthy.

What the script reveals

Here is example output from a congested server:

mean=1.42s p50=0.61s p95=6.83s
exponential p95=4.19s
ratio=1.63

Your numbers will differ. The ratio is the signal.

I used MonkeyCode's free server option as my target. The ratio was above 1.5. The tail was heavy. The mean was fine. The p95 was not.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Setting timeouts with the shape

Stop using a flat timeout. Use the p95 from your distribution. Add a buffer.

If your p95 is 6.8 seconds, set your timeout to 10 seconds. Not 3. Not 30. The shape tells you the right number.

Detecting degradation early

Run this daily. Track the ratio. When it climbs, your server is getting slower. Investigate before users complain.

A rising ratio is an early warning. A falling ratio means the server is healthy.

Limitations

This is not a load test. It tells you the current state, not the capacity. For capacity, use a proper load test.

The exponential fit is a simplification. Real servers have multiple workers, network jitter, and model variability. Use the ratio as a heuristic, not a law.

The sample size matters. 200 requests is a minimum. Use 500 if you can. Small samples produce noisy ratios.

Who should skip this

Teams with a paid SLA have better telemetry. Skip this. Single-user tools with low traffic do not need it. If your request rate is tiny, the queue is empty. The shape is not useful.

Average latency is comfortable. It is also useless. The shape of your latency is the truth. Run the script. Look at the ratio. Then set your timeout like an engineer.

Have you measured your free model server's shape? Share your ratio in the comments.