Stop Timing the First Token: A Streaming FAQ for Free Model Servers

Your chat UI feels instant. Your batch job crawls. Same model. Same server.

Streaming is usually the difference. Streaming changes what your client sees first. It does not change how fast the model thinks.

Most teams time one number: the first token. That number explains maybe a third of the total wait. The rest hides in the gaps between tokens.

This is a myth-busting FAQ. Claims first. Evidence second. Corrected mental model third. Run the probe at the end and check me.

Myth 1: Streaming makes the request faster

The claim: "Set stream=True and the whole call speeds up."

The evidence: Total time stays roughly flat. Streaming moves the first byte earlier. The generation work is identical. Clients feel speed. Stopwatches do not.

The corrected model: Streaming is a perception tool. It is not a throughput tool. Use it where a human waits. Drop it where a pipeline measures.

Myth 2: First-token latency is the only metric that matters

The claim: "My SLA is p95 TTFT. Everything else is noise."

The evidence: Total response time has two parts. First comes the first token. Then comes the stream after it. A 20-token answer? TTFT owns the budget. A 500-token answer? The gaps own it. Tracking one number hides the second half.

The corrected model: Track three numbers. TTFT, median inter-token gap, total time. Which one you optimize depends on where the user waits.

Myth 3: More chunks means a faster-feeling response

The claim: "We flush one chunk per token. The UI feels alive."

The evidence: Every chunk costs a network trip, a JSON parse, and a UI update. Tiny chunks serialize badly under load. Many free servers batch tokens anyway. Chunk count and throughput are different numbers.

The corrected model: Measure tokens per second. Not chunks per second. Sometimes buffering two or three tokens feels smoother than spraying them one by one.

Myth 4: An open stream can't time out

The claim: "I skip timeouts. The stream keeps the connection alive."

The evidence: Stream timeouts are idle-based. A server that emits a token every 25 seconds kills a 15-second idle timeout. Worse: a long stream holds a connection-pool slot. Other requests queue behind your free stream. A quiet jam.

The corrected model: Time out on the gap. Not the connection. If the provider sends keep-alive pings, measure them. Separate connection idle from generation active.

Myth 5: Aborting a stream saves tokens

The claim: "The user clicks stop. We close the socket. Money saved."

The evidence: Closing your socket does not stop the server's loop. Some providers finish generating and bill the orphaned tail. Others check abort signals between chunks, not inside them. Your client stopped. The server did not.

The corrected model: Treat abort as UX. Not cost control. Cap generation server-side with max_tokens. Then compare what the API reported as usage. Assume nothing.

---

The probe: measure your own endpoint

This script runs the same prompt in both modes. It prints median TTFT, median total time, and median token gap. You need Python 3 and the requests package. Roughly ten minutes per endpoint.

import statistics
import time

import requests

ENDPOINT = "https://your-endpoint/v1/chat/completions"
MODEL = "your-model"
HEADERS = {"Authorization": "Bearer $OPENAI_API_KEY"}
PROMPT = "Explain, in 150 words, why free-tier LLM latency is noisy."
TRIALS = 5


def measure(stream: bool) -> dict:
    payload = {
        "model": MODEL,
        "messages": [{"role": "user", "content": PROMPT}],
        "max_tokens": 300,
        "stream": stream,
    }
    started = time.perf_counter()
    first = None
    gaps = []
    last = started

    resp = requests.post(ENDPOINT, json=payload, headers=HEADERS, stream=True)

    if not stream:
        # Time until the first byte, then read the body.
        for _ in resp.iter_content(chunk_size=1):
            first = time.perf_counter() - started
            break
        resp.json()
        total = time.perf_counter() - started
        return {"ttft": first, "total": total, "gap_median": None}

    for line in resp.iter_lines():
        if not line or not line.startswith(b"data:"):
            continue
        if line[5:].strip() == b"[DONE]":
            break
        now = time.perf_counter()
        if first is None:
            first = now - started
        else:
            gaps.append(now - last)
        last = now

    total = time.perf_counter() - started
    return {
        "ttft": first,
        "total": total,
        "gap_median": statistics.median(gaps) if gaps else None,
    }


for stream in (False, True):
    rows = [measure(stream) for _ in range(TRIALS)]
    label = "stream" if stream else "plain"
    print(f"--- {label} ({TRIALS} trials) ---")
    print(f"median TTFT : {statistics.median(r['ttft'] for r in rows):.2f}s")
    print(f"median total: {statistics.median(r['total'] for r in rows):.2f}s")
    gaps = [r["gap_median"] for r in rows if r["gap_median"]]
    if gaps:
        print(f"median gap  : {statistics.median(gaps):.2f}s")

If a provider batches several tokens per chunk, gap_median measures chunks, not tokens. That is still the number your UI sees. So it is the honest metric.

An output shape looks like this. Illustrative, not a benchmark. Run it on your endpoint first.

metric	plain	stream
median TTFT	0.9s	0.4s
median total	3.8s	3.9s
median token gap	—	0.11s

Notice the pattern. First token gets faster. Total time stays flat. That is the whole argument in one table.

When streaming wins (and when it does not)

Your user	TTFT matters?	Total matters?	Verdict
Interactive chat	Yes	No	Stream
Batch summarizer	No	Yes	Plain
Unit test / contract check	Yes	Yes	Stream, then assert total
Proxy fanning out many calls	No	Yes	Plain or buffer chunks

My rule of thumb: keep pipelines plain. Stream only the UI-facing path.

Where I run this probe

I point the probe at MonkeyCode's free model access through the free server option first. Free tiers deserve measurement before trust. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The script is provider-blind. Any OpenAI-compatible endpoint works.

Who should not use this probe

• You are quota-starved. Ten calls can eat a day of free-tier budget. Set TRIALS = 2.
• Your provider has no SSE support. The stream path will hang. Test one call first.
• You need server-side metrics. This is a client-side ruler. Use the provider dashboard for the other half.

Limitations

Five trials is a sample, not a census. Free servers are noisy by design. Raise TRIALS to twenty when your budget allows. Results do not transfer between regions, models, or hours. Rerun on your own endpoint. Never cite the table above as a benchmark. It is a shape, not proof.

Next time a new free model drops, run this probe before you wire it in. The gap between the first token and the last one will surprise you.

Your users wait for the last token. So should your dashboards.