I Kept the CDF. The Mean Clocked Out.

The mean never caught the stall I measured. I kept the CDF instead of it.

One average can hide a whole outage today. You already know that ugly fact. Do you still paste the mean into Slack?

I did that. Then a retry wrote the real story.

I was timing a small coding-assistant HTTP client loop. Not a cluster and not a formal lab. Just a script that sent prompts and waited. The sticky note said two hundred milliseconds even. The room still felt much slower than that. Why did those two clocks disagree so hard?

The mean is a tourist with a short itinerary. It visits the easy hours right after lunch. It leaves before the tail even sits down.

So I threw out the single cheerful number. I kept every sample in a simple list. Then I drew a CDF in the terminal.

The trap I walked into first

I logged only total wall time per call. I averaged the blob without shame. I shipped a screenshot with a tiny smile.

That screenshot lied with a perfectly straight face. P50 sat near the mean like a cousin. P99 lived on another planet entirely. Did I even record time to first byte?

No. I recorded a hug from start to end. Handshake, queue, tokens, and JSON parse. One blob. Completely useless for a later argument.

I wanted shape from this round of timing. Not another glossy dashboard panel. Shape you can read from a chair.

The fixture, not a fairy tale

I did not point this harness at production traffic. I built a local liar on purpose. The server sleeps a little on most calls. Sometimes it sleeps a lot instead.

That is the whole joke of the fixture. Your model endpoint does this under load too. Queue. Cold start. Retry. Name the goblin after you see the cliff.

Label this clearly for anyone reading along. These numbers come from localhost only. They teach a shape. They are not a vendor benchmark.

# cdf_fixture.py
# Local bimodal latency fixture. Not a product claim.
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import random
import time

class Handler(BaseHTTPRequestHandler):
    def log_message(self, format, *args):
        return

    def do_POST(self):
        n = int(self.headers.get("Content-Length", "0"))
        _ = self.rfile.read(n)
        # Most calls are cheap. A few are not.
        if random.random() < 0.08:
            time.sleep(random.uniform(0.35, 0.90))
        else:
            time.sleep(random.uniform(0.04, 0.09))
        body = b'{"ok":true,"n":16}'
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

if __name__ == "__main__":
    ThreadingHTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

Run it and leave it in a corner quietly.

python cdf_fixture.py

The client that keeps every sample

Averages die inside a fold. A list does not die. I wrote a client that stores every latency sample. Then it prints a tiny CDF.

No pandas. No hosted dashboard in this note. Just ranks you can argue with.

# cdf_client.py
import json
import time
import urllib.request

URL = "http://127.0.0.1:8765/work"
N = 200

def one():
    payload = json.dumps({"prompt": "summarize this diff"}).encode()
    req = urllib.request.Request(URL, data=payload, method="POST")
    req.add_header("Content-Type", "application/json")
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=5) as resp:
        _ = resp.read()
    return (time.perf_counter() - t0) * 1000.0

def cdf(samples):
    xs = sorted(samples)
    n = len(xs)
    marks = [50, 90, 95, 99, 100]
    out = []
    for p in marks:
        i = min(n - 1, max(0, int(round(p / 100.0 * n)) - 1))
        if p == 100:
            i = n - 1
        out.append((p, xs[i]))
    return out, xs

if __name__ == "__main__":
    for _ in range(20):
        one()
    samples = [one() for _ in range(N)]
    mean = sum(samples) / len(samples)
    table, xs = cdf(samples)
    print(f"n={len(samples)} mean={mean:.1f}ms min={xs[0]:.1f} max={xs[-1]:.1f}")
    for p, v in table:
        bar = "#" * max(1, int(v / 10))
        print(f"p{p:>3} {v:7.1f}ms {bar}")

Run that against the fixture on the same machine.

python cdf_client.py

You will see a mean that looks almost polite. Then p99 walks in very late. Same machine. Same code. Different story.

Does that surprise you after all these years? It should not surprise you now. It still does, every single time.

Why eight percent owns the graph

The fixture sends eight percent of calls down the long sleep. That is not a rounding error in review. That is a person staring at a spinner.

Think of a bus schedule that is usually fine. One bus in twelve never arrives on time. Do you quote the average wait to your team?

I printed nearest-rank percentiles because they stay honest enough. Sort the list. Pick a rank. Stop smoothing the pain away.

p = 99
n = len(xs)
i = min(n - 1, max(0, round(p / 100 * n) - 1))

People fight about percentile estimators late at night. Nearest-rank is good for a terminal. If you need Hyndman-Fan later, fine. First keep the samples in order.

The mean is a weighted hug of the easy buses. Eight percent still owns the street. Your users ride the street, not the hug.

The graph I actually kept

I do not keep pretty Grafana permalinks around. I keep the terminal block. Something like this showed up in my scrollback later.

Your numbers will move a bit. The cliff should not move.

n=200 mean=112.4ms min=41.2 max=887.0
p 50    68.1ms ######
p 90    84.4ms ########
p 95   402.7ms ########################################
p 99   741.0ms ##########################################################################
p100   887.0ms ########################################################################################

Those bars are from one local run of this fixture. Treat them as a postcard, not a contract. Look at p50 sitting near the mean. Look at p95 falling off a cliff. Would you still ship the mean after that?

I kept that block in the note. I deleted the average from the same note. The cliff is the product you actually shipped. Users do not experience means at all. They experience the bus that never came.

Clocks, warmup, and other small lies

I used time.perf_counter on purpose here. time.time can jump without asking you. NTP is not your friend during a microbench. Have you ever watched a negative duration appear?

Warm the client before you trust the bins. The first calls pay import cost. They pay DNS cache fills. They pay code objects getting hot.

for _ in range(20):
    one()
samples = [one() for _ in range(N)]

Throw those twenty away without mourning them. They are not your distribution today. They are the kettle coming up to temperature.

Garbage collection can steal a bin too. If p99 spikes once, look at GC. A CDF with one poisoned sample still talks. It may tell the wrong story though.

I also printed a crude hash bar per percentile. That is not science class. It is a wall I can see from a chair.

First byte, then the rest of the body

I stopped asking if it was fast. I asked who was waiting on me. I added a second clock for first byte. Not the full body. First byte.

def one_ttfb():
    payload = json.dumps({"prompt": "summarize this diff"}).encode()
    req = urllib.request.Request(URL, data=payload, method="POST")
    req.add_header("Content-Type", "application/json")
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=5) as resp:
        ttfb = (time.perf_counter() - t0) * 1000.0
        _ = resp.read()
        total = (time.perf_counter() - t0) * 1000.0
    return ttfb, total

On this fixture they move together closely. That is the point of the liar. On a real stream they split apart. First byte can look just fine. The rest can crawl home slowly. Then you know which hallway to walk at night.

I also toggled connection reuse on the client. urllib can hide a cold socket. Want a nastier tail in the CDF? Force a new connection every time.

import http.client

def one_cold():
    payload = json.dumps({"prompt": "summarize this diff"}).encode()
    t0 = time.perf_counter()
    conn = http.client.HTTPConnection("127.0.0.1", 8765, timeout=5)
    conn.request(
        "POST",
        "/work",
        body=payload,
        headers={"Content-Type": "application/json"},
    )
    resp = conn.getresponse()
    _ = resp.read()
    conn.close()
    return (time.perf_counter() - t0) * 1000.0

Cold sockets add jitter even on localhost today. On a remote name they add DNS lookups. They add TLS. They add a handshake you never budgeted for. The mean still shrugs at you. The CDF does not shrug.

Change one knob. Keep the new postcard. That is the whole experiment.

Where a free model and free server fit

I needed a quiet box and a second pair of eyes. Not a chorus. A scratch loop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode's free model access on the CDF printer. I used the free server option to run the fixture. That is the whole cameo in this note. The harness still works if you delete those sentences.

The model did not invent the bins for me. I still ranked every sample by hand. You should rank them too. Do not let a chat window average your night.

What this will not save you from

This is not load testing in any serious sense. Two hundred sequential calls are only a sketch. They are not a soak or a region test. They still are not your written production SLO.

Do not use a localhost CDF to pick a vendor. Do not paste these bars into a pricing fight. Do not trust a mean from a warm laptop either.

Skip this if traces already give you real histograms. Skip this if you only need token clocks. Skip this when you cannot steer the server sleep. A CDF of pure noise is still noise.

Also skip it if you will not keep raw samples. Binned vanity is just a mean in a costume. People are arguing about tests for models again. Fine. Start with the client you actually ship. If your timer is a single average, the debate is already lost.

The rule I taped to the monitor

If you can only keep one picture, keep the tail. Ask the mean to sit down for a minute. Ask p99 who hurt it last.

Then change one thing only after that. Reuse the socket. Cut the retry storm. Drop a giant tool schema. Run the client again after the change. Keep the new CDF beside the old one.

Did the cliff move after the change? Good. You learned a real thing. Did only the mean move a little? You learned nothing useful tonight.

I still catch myself averaging on busy days. Then I look at that old terminal block again. The mean had clocked out early. The CDF stayed for the whole shift.