Nginx proxy_buffering Ate My LLM Stream: 11s to First Token

On my laptop the report streamed in like a little typewriter. Token by token, 0.4 seconds to the first character, exactly the experience I wanted.

Then I deployed it. In production the page sat there doing absolutely nothing for eleven seconds, and then slammed all 1,800 tokens into the DOM in a single frame. Same code. Same model. Same prompt.

I spent three days blaming the model. It was nginx proxy_buffering. And behind nginx there were three more layers doing the same thing, each one politely holding my tokens hostage for my own good.

TL;DR

• If your LLM stream arrives all at once in production but streams fine locally, a proxy is buffering it. proxy_buffering on is the nginx default and it will happily collect your entire SSE response before forwarding one byte.
• Turning off proxy_buffering alone barely helped me: 11.4s to 9.1s. gzip was the bigger dam. Compression buffers your event stream too.
• The full fix was four layers: nginx buffering, gzip, CDN transform, and my own token-chunking code. Final result: 0.7s to first visible token.
• Diagnose it in one command, not three days: curl -N through the proxy, then curl -N straight at the origin port. If the origin is fast and the proxy is slow, stop reading your Python.
• Bonus bug found on the way: proxy_read_timeout defaults to 60s, which silently truncated every report that ran longer than a minute.

What was actually streaming here?

The system is the written-report step of Preterview, an interview prep platform I run (full disclosure: I built it). It runs a realistic voice interview, then generates a scored written report from the transcript. That report is 1,500 to 2,000 tokens of markdown, which takes long enough to generate that streaming isn't a nicety, it's the difference between "thinking" and "broken."

The stack is boring on purpose: FastAPI + StreamingResponse, Server-Sent Events, uvicorn on 127.0.0.1:8000, nginx in front, a CDN in front of that. Nothing exotic. That's the point. Every layer in that list buffers by default, and every one of them thinks it's helping.

Here's what a dead stream cost me in real user behavior: in the week before I fixed it, 41 of 120 report sessions had a page reload before the report finished. People assumed it had hung, because from the browser's point of view it had. Each reload kicked off another generation. I was paying twice to deliver a worse experience.

Why does nginx proxy_buffering break SSE token streaming?

Because nginx proxy_buffering exists to protect your app server from slow clients. It reads the upstream response as fast as the upstream can produce it, parks it in memory (and then on disk), and drips it out to the client at the client's pace. For a 400KB JSON payload that's genuinely good engineering. Your Python worker gets freed immediately instead of babysitting someone on hotel wifi.

For SSE it's a catastrophe, because the value of the response is entirely in its timing. nginx doesn't know that. It sees bytes. It fills a 4k buffer, and only when that buffer is full (or the upstream closes the connection) does the client see anything.

So with proxy_buffering on, the "stream" becomes: generate for eleven seconds, buffer, flush once. Which is exactly the shape of the bug I saw.

The two fixes:

location /api/report/stream {
    proxy_pass http://127.0.0.1:8000;
    proxy_http_version 1.1;
    proxy_set_header Connection '';

    proxy_buffering off;
    proxy_cache off;
    gzip off;

    proxy_read_timeout 3600s;
    chunked_transfer_encoding off;
}

Or, better if you don't want to touch nginx config for every new endpoint, let the app declare it per response:

return StreamingResponse(
    token_generator(),
    media_type="text/event-stream",
    headers={
        "Cache-Control": "no-cache, no-transform",
        "X-Accel-Buffering": "no",
        "Connection": "keep-alive",
    },
)

X-Accel-Buffering: no is nginx's own opt-out header, and it's the one I'd reach for first. It travels with the response, so a new streaming route works without a deploy of your proxy config.

I lost an afternoon on this header because I set it on the wrong thing. I emitted it inside the first SSE event instead of on the HTTP response headers. It arrived as part of the body, nginx never saw it as a header, and the stream stayed dead. Headers go on the StreamingResponse, not in the generator.

What were the four layers, and what did each one cost?

Four things buffered my tokens. Here's the measured time to first visible token in the browser, on the same 1,800-token report, fixing one layer at a time:

Stage	First visible token
Production baseline	11.4s
proxy_buffering off	9.1s
gzip excluded for text/event-stream	3.2s
CDN: correct content-type + no-transform	1.9s
My own chunking removed	0.7s
(Local dev, for reference)	0.4s

Layer 1: nginx proxy_buffering. 11.4s to 9.1s. A 2.3 second improvement on a problem I was certain was 100% nginx. That gap is the whole lesson of this post: I almost reverted the fix because it "didn't work."

Layer 2: gzip. This is the one that gets people. gzip has to accumulate input before it can emit a compressed block, so a compressor sitting in your response path is a buffer whether or not you disabled the other buffer. text/event-stream is not in nginx's default gzip_types, so you'd think you're safe. I wasn't, because I'd added a wide gzip_types list years earlier and copied it forward into every server block since. Check yours before you assume. Removing it: 9.1s to 3.2s.

Layer 3: the CDN. Anything that can transform your response body can hold it. In my case the stream was leaving my origin with a content-type the edge didn't recognize as streamable, because I was setting media_type on an inner response object and the outer one defaulted to application/json. Setting text/event-stream correctly and sending Cache-Control: no-cache, no-transform took it to 1.9s. If you're not sure, bypass the CDN with a direct DNS entry to your origin and measure again. Don't theorize about edge behavior, measure it.

Layer 4: me. This was the embarrassing one. My generator wasn't yielding tokens, it was accumulating them until a "markdown-safe boundary" — a closed **, a finished list item — so the UI never flashed a half-rendered bold marker. That felt clever. In practice it held 40 to 80 tokens at a time, and inside a heading or a fenced code block it held far longer. I was the fourth proxy in my own stack. I ripped it out and made the renderer tolerant of partial markdown instead, which is where that logic belonged. 1.9s to 0.7s.

How do you find which layer is buffering your stream?

Bisect at the proxy boundary, with two curl calls. This takes about ninety seconds and would have saved me three days.

# through the proxy, from outside
curl -N -s -o /dev/null -w 'ttfb %{time_starttransfer}s\n' \
  https://api.example.com/api/report/stream

# straight at the origin, on the box, skipping nginx entirely
curl -N -s -o /dev/null -w 'ttfb %{time_starttransfer}s\n' \
  http://127.0.0.1:8000/api/report/stream

time_starttransfer is time to first byte, which for a stream is the only number that matters. -N disables curl's own output buffering, and forgetting it is the classic false positive: curl will happily make a perfectly healthy stream look broken.

My two numbers were 0.5s at the origin and 11.4s through the proxy. That single comparison rules out the model, your prompt, your token generator, your event loop, and your frontend, all at once. If the origin is fast, no amount of staring at your Python is going to help you.

For the layers above nginx, keep bisecting the same way: hit the origin's public IP directly to skip the CDN, then compare with the CDN in path.

Should you just turn proxy_buffering off everywhere?

No, and I tried, which is how I learned why not. I put proxy_buffering off in the http block because it was one line instead of many, and file uploads and large JSON responses got measurably worse under concurrency. That's the feature working as designed: with buffering off, a slow client's pace becomes your app worker's pace, and your workers spend their time trickling bytes to phones on bad connections instead of serving requests.

Scope it to the streaming location, or use X-Accel-Buffering: no per response. Buffering is the right default for almost every route you have. It's wrong for exactly the routes where time-to-first-byte is the product.

One more thing to set while you're in that block: proxy_read_timeout defaults to 60 seconds, measured between reads from upstream. Long report generations hit it and the connection closed at almost exactly 60.0s, mid-sentence, with no error anywhere in my application logs. Six of those 120 sessions had silently truncated reports and I'd never noticed, because a truncated markdown report still looks like a report.

So what actually fixes an LLM stream that arrives all at once?

If your tokens stream locally and arrive in one lump in production, the cause is buffering between your app and the browser, not your model or your generator. Set proxy_buffering off (or send X-Accel-Buffering: no) on the streaming route, disable gzip for text/event-stream, send Content-Type: text/event-stream with Cache-Control: no-cache, no-transform so your CDN passes it through untransformed, raise proxy_read_timeout well past your longest generation, and then check whether your own code is batching tokens before it yields them. Confirm each layer with curl -N through the proxy versus straight at the origin. Four layers, 11.4 seconds to 0.7, and three of the four were defaults I never chose.

---

Written by the developer behind Preterview, an interview prep platform.