I Chased Tokens Per Second. First Token Won.

Tokens per second looks like a clean scientific score. It often hides the stall your users actually feel. I stopped averaging the whole stream this week.

Why did the demo feel snappy while the chart looked sad? Why did the next prompt reverse that story? The wait sat between accept and first token.

I wanted a graph I could actually keep. I did not want another standup wallpaper chart. I wanted a plot that survived a skeptical teammate.

So I treated streaming like a packet capture. I timed the wait and every later chunk. The method stays boring on purpose, and that helps.

A long completion can post a fat throughput number. The user still stared at a blank bubble. Did your pretty score measure work, or waiting?

I ran this as a local lab notebook. The numbers below are local script fixtures only. Please treat them as examples, not production truth.

The bottleneck I actually wanted

I used to log total elapsed time. Then I divided elapsed time by chunk count. That ratio feels like honest throughput.

That ratio is still a blended cocktail, though. Queue delay, warmup, and sampler loops hide inside it. One slow first token poisons the sip.

Later chunks can sprint and still look heroic. The average never files a complaint. Want a better question for the next incident?

Ask when the first byte actually moved. Then ask whether later bytes arrived steadily. Those two clocks split stall from engine.

I kept a heatmap of inter-chunk gaps. The average line was a polite little liar. A single long hole felt like a freeze.

Users do not experience means on a chat bubble. They experience the hole in the drip. That hole is the whole argument here.

A tiny probe you can rerun

This probe is labeled as a local experiment. It records start, first token, and finish. It also records gaps between streamed chunks.

I used a plain HTTP streaming client. No vendor SDK sat in that path. Point it at any OpenAI-style chat endpoint.

# stream_probe.py
# Labeled experiment: local clocks, not a vendor benchmark.

import json
import os
import sys
import time
import urllib.request


def percentile(values, p):
    if not values:
        return None
    ordered = sorted(values)
    idx = min(len(ordered) - 1, int(round((p / 100.0) * (len(ordered) - 1))))
    return ordered[idx]


def iter_sse_lines(resp):
    buf = b""
    while True:
        piece = resp.read(256)
        if not piece:
            if buf:
                yield buf
            break
        buf += piece
        while b"\n" in buf:
            line, buf = buf.split(b"\n", 1)
            yield line


def probe(url, payload, timeout=120):
    data = json.dumps(payload).encode("utf-8")
    headers = {"Content-Type": "application/json"}
    token = os.environ.get("API_TOKEN")
    if token:
        headers["Authorization"] = "Bearer " + token
    req = urllib.request.Request(url, data=data, headers=headers, method="POST")
    t0 = time.perf_counter()
    first = None
    chunks = 0
    gaps = []
    last = t0
    text = []

    with urllib.request.urlopen(req, timeout=timeout) as resp:
        for raw in iter_sse_lines(resp):
            now = time.perf_counter()
            line = raw.decode("utf-8", errors="replace").strip()
            if not line.startswith("data:"):
                continue
            body = line[5:].strip()
            if body == "[DONE]":
                break
            try:
                obj = json.loads(body)
            except json.JSONDecodeError:
                continue
            delta = (
                obj.get("choices", [{}])[0]
                .get("delta", {})
                .get("content")
            )
            if not delta:
                continue
            if first is None:
                first = now
            gaps.append(now - last)
            last = now
            chunks += 1
            text.append(delta)

    end = time.perf_counter()
    ttft = None if first is None else (first - t0)
    gen = 0.0 if first is None else (end - first)
    tps = 0.0 if gen <= 0 else (chunks / gen)
    later = gaps[1:]
    return {
        "ttft_s": ttft,
        "total_s": end - t0,
        "chunks": chunks,
        "tps_after_first": tps,
        "gap_p50_s": percentile(later, 50),
        "gap_p99_s": percentile(later, 99),
        "chars": sum(len(x) for x in text),
        "gaps_s": later,
    }


if __name__ == "__main__":
    url = sys.argv[1]
    prompt = sys.argv[2] if len(sys.argv) > 2 else "Explain tail latency in one paragraph."
    payload = {
        "model": os.environ.get("MODEL", "local"),
        "messages": [{"role": "user", "content": prompt}],
        "stream": True,
        "max_tokens": 256,
    }
    result = probe(url, payload)
    printable = {k: v for k, v in result.items() if k != "gaps_s"}
    print(json.dumps(printable, indent=2))

Save it as stream_probe.py and run it from a shell. You should use the same prompt every time. Change one variable per run, or you will learn noise.

python3 stream_probe.py http://127.0.0.1:8080/v1/chat/completions
python3 stream_probe.py http://127.0.0.1:8080/v1/chat/completions "Count from 1 to 40."

Warm the process with a throwaway prompt first. Then capture five runs of the real prompt. Discarding the cold start keeps the graph honest.

Sample fixture output from that local run follows. Do not read it as a product claim. Do not paste it into a vendor table.

{
  "ttft_s": 1.82,
  "total_s": 4.11,
  "chunks": 94,
  "tps_after_first": 41.0,
  "gap_p50_s": 0.018,
  "gap_p99_s": 0.240,
  "chars": 612
}

See the split in those three clocks? Throughput after first token looks perfectly fine. The user still paid almost two silent seconds.

Would you ship that graph to a product manager? I would not ship that blended score anyway. I would ship those two split clocks instead.

The experiment I kept

I repeated the same prompt five times. I discarded the first run as a warmup. I plotted TTFT right beside gap P99.

Throughput wandered around and taught me almost nothing. TTFT explained the thinking pause in the bubble. Gap P99 explained that mid-sentence freeze well.

I also logged chunk sizes on the side. Some servers drip one token at a time. Some servers dump a whole phrase at once.

If your client paints on every chunk, burst still stutters. The wire itself can still be perfectly fine. Then the renderer gets bored and then flooded.

That is why I refuse a single score now. A single score is just a blender. You cannot unmix waiting from working later on.

I wrote the five runs into a CSV. Then I sketched TTFT against gap P99 next. That pair is the graph I still have open.

# keep_graph.py
# Proposal: turn probe JSON lines into a tiny CSV.

import json
import sys

print("run,ttft_s,gap_p99_s,tps_after_first,chunks")
for i, line in enumerate(sys.stdin, 1):
    line = line.strip()
    if not line:
        continue
    row = json.loads(line)
    print(
        f"{i},{row['ttft_s']:.3f},{row['gap_p99_s']:.3f},"
        f"{row['tps_after_first']:.1f},{row['chunks']}"
    )

Pipe the probe into it after each run. Please do not average those ugly rows away. The ugly run is the one users will feel.

python3 stream_probe.py http://127.0.0.1:8080/v1/chat/completions >> runs.jsonl
python3 keep_graph.py < runs.jsonl

Look at run three before you trust run two. If TTFT jumps while later gaps stay flat, the engine was waiting, not writing. If later gaps jump while TTFT stays calm, the stall moved into the stream.

That split is the whole profile. Everything else is interior decoration. Keep the pair that argues with the demo.

Where a nearby free endpoint helps

I needed a cheap loop for the probe. Paid APIs punish retries and those broken parsers. You want to break the client on purpose.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I pointed the probe at MonkeyCode for that loop. Free model access and a free server option kept the retries local. I am not naming models, quotas, or hardware here.

I am naming a setup I could restart without paperwork. The clocks remain the subject, not the vendor. If you already have a compatible URL, the same script works.

If you need a nearby loop like that, start with that free setup. Then keep the TTFT plot instead of a throughput trophy.

What the heatmap taught me

I dumped gaps into a histogram in the terminal. I skipped Grafana and the twelve-panel shrine. A mean line would have clapped for the wrong room.

def ascii_hist(gaps, width=40):
    if not gaps:
        return
    lo, hi = min(gaps), max(gaps)
    span = max(hi - lo, 1e-9)
    buckets = [0] * 10
    for g in gaps:
        i = min(9, int((g - lo) / span * 10))
        buckets[i] += 1
    peak = max(buckets) or 1
    for i, n in enumerate(buckets):
        bar = "#" * int(n / peak * width)
        edge = lo + (span / 10.0) * i
        print(f"{edge * 1000:7.1f} ms | {bar}")

Call it on the later gaps after each run. Skip the first gap if you already print TTFT. The first gap is the wait you already named.

The mass sat near twenty milliseconds in my fixture. Then a lonely tail jumped past two hundred. That lonely tail is the freeze users see.

The mean never voted the tail into office. I used to worship those total time bars. This week I kept the gap histogram instead.

Is the model slow, or is the stream polite then silent? That question is the whole debugging job here. Those simple throughput averages cannot answer that question.

What I changed after the plot

I stopped printing tokens per second in the demo script. I printed TTFT and gap P99 beside the answer. The room talked about those two clocks immediately.

I also delayed first paint until the first real chunk. A spinning glyph for 1.8 seconds felt honest. A frozen empty bubble just felt broken.

On the server side I looked at write flushing next. A buffered SSE stream merges polite tokens into late bursts. Your client then draws a stall that the model did not cause.

None of that work required a new model. It required a graph that argued with me. The probe is a flashlight, not a leaderboard.

I also stopped comparing cold runs with warm runs. Mixing them rebuilds the blender I just smashed. Cold start belongs on its own axis, beside queue depth.

If you only change the prompt, say so on the plot. If you only change the client flush, say that too. Silent legend changes are how throughput becomes folklore.

Limitations, because the script is rude

This probe counts chunks, not true tokenizer tokens. A chunk may hold a word or a clause. Do not publish it as paper-grade throughput work.

It also ignores TLS, DNS, and proxy buffers. A nearby server still hides those extra costs. A public edge will not hide them at all.

HTTP line buffering can merge events on you. Your P99 gap may be the codec itself. Flush settings and runtime buffering both still matter.

Who should skip this whole trick today? Anyone shipping an SLO from one laptop run should stop. Anyone comparing vendors with this tiny fixture should stop too.

Skip it for token-accurate billing math as well. Also skip it if your product is pure batch. If you only need a file, first-token wait is vanity.

Measure job time and failure rate instead here. Do not use it to crown a free server as faster. Localhost quietly removes the ocean sitting in between.

That ocean is where your real users live. A local heatmap is a rehearsal, not opening night. Bring the same clocks to a client near those users.

The graph I still have open

I did not keep the old throughput chart. I kept TTFT versus gap P99 for one prompt. When TTFT jumps, I look at cold start and queue.

When gap P99 jumps, I look at sampler and write path. I re-run the probe after every quick fix. If the first token does not move, you optimized the wrong room.

If throughput soars and the bubble still blinks, you already know why. Please keep the graph that argues with you. Then throw away the one that claps.