The Stall Was in the Stream, Not the Model

Your p50 looks fine. Your users still say it feels slow. Why?

Because the average hides the pause. A model can answer in two seconds and still feel broken. The user watches text stop, then start, then stop again.

That pause has a name. It is the inter-token gap. Almost nobody measures it.

Time to first token is a comfortable lie

TTFT tells you when the first chunk arrives. It says nothing about the ninety chunks after it. You can ship a great TTFT and a terrible experience.

Total latency is the other comfortable lie. It folds queueing, generation, and network into one number. When it moves, you learn nothing about where.

So split the clock into three parts. Wait before chunk one. Gaps between chunks. Time after the last chunk. Now each part has an owner.

A harness with no dependencies

I want this to run anywhere, including a small box. So it uses only the standard library. No SDK, no virtualenv gymnastics, no build step.

Save this as gaps.py. It is a harness, not a result. I have not run it against your endpoint, and I will not hand you numbers I did not measure.

import json, os, statistics, sys, time, urllib.request

BASE  = os.environ["MC_BASE_URL"].rstrip("/")
KEY   = os.environ["MC_API_KEY"]
MODEL = os.environ["MC_MODEL"]

def stream_once(prompt, stall_ms=250):
    body = json.dumps({
        "model": MODEL,
        "stream": True,
        "messages": [{"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(
        BASE + "/chat/completions", data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer " + KEY},
    )
    t0 = time.perf_counter()
    marks, ttft = [], None
    with urllib.request.urlopen(req, timeout=120) as r:
        for raw in r:
            line = raw.decode("utf-8", "replace").strip()
            if not line.startswith("data:"):
                continue
            payload = line[5:].strip()
            if payload == "[DONE]":
                break
            obj = json.loads(payload)
            delta = obj["choices"][0].get("delta", {})
            if not delta.get("content"):
                continue
            now = time.perf_counter()
            if ttft is None:
                ttft = now - t0
            marks.append(now)
    total = time.perf_counter() - t0
    gaps = [(b - a) * 1000 for a, b in zip(marks, marks[1:])]
    stalls = [g for g in gaps if g >= stall_ms]
    return {"ttft_ms": (ttft or 0) * 1000, "total_ms": total * 1000,
            "chunks": len(marks), "stalls": len(stalls),
            "gap_p50": statistics.median(gaps) if gaps else 0,
            "gap_p95": sorted(gaps)[int(len(gaps) * 0.95)] if gaps else 0,
            "gap_max": max(gaps) if gaps else 0}

def main():
    prompt = open(sys.argv[1]).read()
    runs = int(sys.argv[2]) if len(sys.argv) > 2 else 20
    for i in range(2):                      # warmup, discarded
        stream_once(prompt)
    rows = [stream_once(prompt) for _ in range(runs)]
    for k in rows[0]:
        vals = [r[k] for r in rows]
        print(f"{k:10s} p50={statistics.median(vals):8.1f} max={max(vals):8.1f}")

main()

Run it like this. Adjust the variables to whatever your endpoint hands you.

MC_BASE_URL="$MC_BASE_URL" MC_API_KEY="$MC_API_KEY" \
MC_MODEL="$MC_MODEL" python3 gaps.py prompt.txt 20

Two warmup calls get dropped on purpose. Cold connections lie about gaps.

What the output actually tells you

Four shapes show up, and each one has a different owner.

TTFT high, gaps tight. The queue is in front of the model. Your input is long, or your concurrency is high, or both.

TTFT tight, gaps fat in the tail. Generation starts fast, then hiccups. Suspect the transport before the model. Proxy buffering is the usual thief here.

Everything flat, total still high. You asked for a long answer. Cap max_tokens and stop blaming infrastructure.

Spread wide across identical runs. Shared capacity, and no amount of averaging will save you. Report p95 or report nothing.

Where the free tier earns its place

Here is the boring part everyone skips. This harness needs twenty real runs per prompt. Do that on a paid key and you start rationing experiments. Rationing experiments is how you end up with n=1 again.

MonkeyCode offers free model access and a free server option, which is what makes repeated runs cheap enough to be routine. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I keep the allowance vague on purpose. Check the project's current page for the live terms before you plan a batch. Allowances move, and a number quoted in a blog post is stale by the time you read it.

The free server matters just as much. I run the collector there, not on my laptop. Same clock, same network path, no coffee-shop wifi in the sample. A measurement host one hop away from your users beats a faster laptop every time.

On a small free box, a stdlib-only script is a feature. No wheels to build, no memory spikes, no surprise dependency on a C compiler.

Start it, point it at one prompt, and let it run while you do something else. That is the whole workflow.

nohup python3 gaps.py prompt.txt 20 > gaps.log 2>&1 &
tail -f gaps.log

The experiment I actually insist on

One prompt is not an experiment. Pick three: a short question, a medium explanation, a long structured answer. Same prompt text every run. Temperature fixed.

Twenty runs each. Randomize which prompt goes first, so a slow window does not attach itself to one input. Keep the raw JSON rows, not just the summary. You will want them when someone asks why p95 jumped.

Then repeat the whole thing from a second host. If the gaps move and TTFT does not, you have proven the problem is transport. That is a one-afternoon result, and it ends a lot of arguments.

Who should not bother

If your product shows nothing until the full answer lands, skip this. Gaps are invisible to you, and total time is the only clock that bills.

If you cannot spare twenty calls per prompt, skip it too. A harness fed with three samples is a rumour with a chart.

And if your SLO is written on the mean, fix the SLO first. You will keep chasing ghosts until the tail is the number that matters.

So what is your slowest chunk? Not your slowest request. The gap you have never plotted.

Give the harness twenty runs on the free server and look at gap_p95 first. That column is where the truth usually lives.