FAQ: Streaming Tokens Is Not Faster Generation — Four Questions, One Stopwatch Test

My chat UI felt slow for a week.

I blamed the model first. That was wrong.

My reverse proxy was holding bytes until the response finished. One config line fixed it.

Here are the four questions I had to answer with a stopwatch.

Q1. Does streaming make generation faster?

No. Streaming changes when tokens arrive, not how fast they are computed.

Total wall time is compute plus transport. Streaming adds framing bytes, so total time can even rise slightly.

What improves is perceived latency. The user sees the first words early.

Corrected mental model: keep two separate budgets.

• TTFT (time to first token) is a responsiveness budget. It matters for chat UIs.
• Total time is a compute budget. It matters for batch jobs.

Ask yourself who is waiting, and what they need to see first.

Q2. My endpoint speaks SSE. So the client sees tokens immediately, right?

Not necessarily. Every hop between process and screen can hold bytes.

Common buffering suspects, in the order I check them:

1. Reverse proxy buffering (proxy_buffering in nginx).
2. Compression on the stream route (gzip likes to fill blocks).
3. A CDN or load balancer that buffers by default.
4. App middleware that wraps the response object.
5. Client libraries that aggregate the whole body before returning it.

A working nginx route for a token stream looks like this:

location /v1/chat/stream {
    proxy_pass               http://127.0.0.1:8081;
    proxy_http_version       1.1;
    proxy_set_header         Connection '';
    proxy_buffering          off;
    proxy_cache              off;
    gzip                     off;
    chunked_transfer_encoding on;
    proxy_read_timeout       300s;
}

If you cannot edit the proxy config, the app can ask nicely. Send X-Accel-Buffering: no as a response header.

That header is an nginx convention, not an HTTP standard. It does nothing on other proxies. Check your vendor docs before trusting it.

I hit this on a small self-hosted box while working with MonkeyCode's free model access and free server option. Long-lived streams are exactly where a shared host's limits become visible.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Q3. Doesn't curl -w '%{time_starttransfer}' give me time to first token?

No. time_starttransfer records the first response byte. On a streamed response, that byte is usually a header.

Headers can land in 40 ms while the first token arrives three seconds later.

Also, curl -N only disables curl's own output buffering. It says nothing about your server or proxy.

So I use a probe that timestamps real chunks instead:

# ttft_probe.py - first chunk and inter-chunk gaps on an SSE endpoint
# Run: python ttft_probe.py http://localhost:8080/v1/chat/stream
import sys, time, requests

url = sys.argv[1]

# Example payload only - adapt the schema to your own API.
payload = {'messages': [{'role': 'user', 'content': 'count to twenty'}]}

t0 = time.perf_counter()
marks = []

with requests.post(url, json=payload, stream=True,
                   headers={'Accept': 'text/event-stream'},
                   timeout=120) as r:
    r.raise_for_status()
    # chunk_size=None yields bytes as the transport delivers them
    for chunk in r.iter_content(chunk_size=None):
        if not chunk:
            continue
        now = time.perf_counter()
        if not marks:
            print(f'first_chunk={now - t0:.3f}s ttft_ms={int((now - t0) * 1000)}')
        marks.append(now)

total = time.perf_counter() - t0
gaps = [b - a for a, b in zip(marks, marks[1:])]
print(f'chunks={len(marks)} total={total:.3f}s')
if gaps:
    ordered = sorted(gaps)
    print(f'gap_p50={ordered[len(ordered) // 2] * 1000:.0f}ms '
          f'gap_max={max(gaps) * 1000:.0f}ms')

Run it twice. Once straight at the upstream port. Once through the proxy.

If the direct run shows early chunks and the proxied run shows one big chunk at the end, you have a buffering bug. Not a slow model.

Illustrative numbers from one of my runs. One request, not a benchmark:

Path	First chunk	Chunks seen	Total
direct to app	~0.6 s	many	~7.9 s
proxy, proxy_buffering on	~7.9 s	1	~8.0 s
proxy, proxy_buffering off	~0.7 s	many	~8.0 s

Same model, same prompt, one changed variable.

Notice what the table does not show: total time barely moved. Only the arrival pattern changed.

Q4. Is one chunk one token?

No. Chunk boundaries are a transport detail, not a token boundary.

One chunk can carry five SSE events. One event can span two chunks.

So parse by the protocol, not by the buffer. SSE separates events with a blank line:

# fold transport chunks into complete SSE events
buffer = ''
for chunk in r.iter_content(chunk_size=None, decode_unicode=True):
    buffer += chunk
    while '\n\n' in buffer:
        event, buffer = buffer.split('\n\n', 1)
        for line in event.splitlines():
            if line.startswith('data:'):
                data = line[len('data:'):].strip()
                if data and data != '[DONE]':
                    handle_event(data)  # your handler here

This matters most for accounting. If you count chunks, your token estimates drift quietly.

Count parsed events instead. Or read the usage block the API returns, when it returns one.

Putting the three numbers back together

Three properties are now separate in my head:

• Compute time is a model property. Measure it upstream, with no proxy in the path.
• Delivery time is a network and proxy property. Measure it at the client.
• Perceived responsiveness is a product decision. Streaming plus a fast first token.

Conflating those three is how I lost a week.

A quick decision table

Situation	Stream?	Why
Chat UI, long answers	Yes	Early feedback beats total time
Batch classification	No	Just collect results; streaming adds code
CLI printing progress	Yes	Humans read while it works
Server-to-server pipeline	Usually no	Buffering bugs cost more than they save
Mobile clients on flaky networks	Test first	Middleboxes may buffer or truncate

Where this workflow fails

My probe measures one path, from one client, at one moment. That is not a benchmark.

It cannot tell you why the model itself was slow. It cannot see inside a CDN you do not control.

It also cannot predict long-connection policy on your host. If idle connections get reaped or request duration is capped, a 300-second stream will surface it loudly.

Read the published limits instead of guessing them.

And do not trust a single sample. Five runs minimum, then look at the spread.

Who should skip this

Skip it if you have a hard latency SLO and no authority to change the proxy. You need platform owners, not a script.

Skip it if responses finish under 300 ms. The extra streaming complexity is not worth it.

Skip it if you cannot safely send a test prompt to the endpoint.

What I keep in the repo now

The probe sits next to the integration tests. It runs in about a minute.

When someone says the model feels slow, I run it before opening a ticket. Nine times out of ten, the answer is a hop, not a model.

If you try this on any free endpoint, run the probe twice — direct and proxied — and paste both numbers into the issue.