If your LLM response streams token-by-token on localhost but lands as a single blob in production, your application code is almost certainly fine. Something between your process and the browser is holding bytes: a compression layer, a reverse proxy with response buffering on, or a platform that buffers the entire response before returning it. The fix is not to change how you write chunks — it is to walk the request path one hop at a time and find which hop stopped forwarding.
I have now debugged this on three different stacks, and every time the instinct was wrong. The first suspect is always the streaming code. It is almost never the streaming code.
Why does streaming work on localhost but not in production?
On localhost there is exactly one hop: your process writes to a socket, the browser reads it. In production there are usually four or five — a load balancer, a reverse proxy, possibly a CDN, maybe a serverless runtime wrapper, and compression middleware inside your own app. Each of those is allowed to accumulate bytes before forwarding, and most of them do it by default because buffering makes normal request/response traffic faster.
Server-Sent Events (SSE) and chunked responses are the exception where buffering destroys the entire point. The bytes still arrive correctly, so nothing errors, no log line fires, and monitoring stays green. You get a silent latency bug: the user waits eight seconds and then sees a wall of text.
The takeaway: buffering is a correctness-preserving optimization, which is exactly why nothing in your stack will warn you about it.
How do I find which hop is buffering?
Bisect the path. Start at the origin process and add one hop per test. This takes about ten minutes and beats guessing.
Test the app directly, on the box, bypassing every proxy:
curl -N --no-buffer -H "Accept: text/event-stream" \
http://127.0.0.1:3000/api/chat
-N disables curl's own output buffering — leave it off and you will misdiagnose your infrastructure because of your test client. If chunks appear one at a time here, your handler is correct and the problem is downstream.
To see arrival timing rather than just final output, stamp each line:
curl -sN https://example.com/api/chat | while IFS= read -r line; do
printf '%s %s\n' "$(date +%T.%N)" "$line"
done
(%N is GNU coreutils; on macOS use gdate from coreutils, or pipe through ts from moreutils.) A healthy stream shows timestamps creeping forward. A buffered one shows every line stamped within the same millisecond at the end — that single observation tells you the response was assembled somewhere and released at once.
Then repeat against each hop: the internal service address, the proxy address, the public hostname. The first URL that produces same-millisecond timestamps is the hop that owns your bug.
| Hop | Typical symptom | Quick check | Usual fix |
|---|---|---|---|
| Compression middleware (in-app) | Buffered until ~1KB accumulates, then bursts | Response has content-encoding: gzip
|
Exclude text/event-stream from the compression filter |
| nginx / reverse proxy | Fully buffered, released at end | Works on origin port, not through proxy |
proxy_buffering off or send X-Accel-Buffering: no
|
| CDN / edge layer | Buffered only on the public hostname | Origin hostname streams, public one does not | Bypass rule for the route; check edge compression |
| Serverless wrapper | Always fully buffered, no config helps | Same behavior everywhere including local emulation | Use a runtime with explicit response-streaming support |
| Browser client | Network tab shows chunks, UI updates once |
curl -N streams fine |
Read the body as a stream, do not await res.text()
|
Which layers buffer most often?
Compression middleware, inside your own app. This is the one people miss because it is not "infrastructure." Express's compression package compresses anything compressible considers text — and text/event-stream is text/*, so it qualifies. gzip needs a block of input before it emits output, so your tokens sit in the compressor. Exclude the content type:
const compression = require("compression");
app.use(
compression({
filter: (req, res) =>
res.getHeader("Content-Type") !== "text/event-stream" &&
compression.filter(req, res),
})
);
The same applies to Starlette/FastAPI's GZipMiddleware — if it is installed, take it out of the path for the streaming route and re-test before touching anything else.
Reverse proxies. nginx buffers proxied responses by default (proxy_buffering on), which is the right default for HTML and the wrong one for SSE. Scope the exception to the route rather than turning it off globally:
location /api/chat {
proxy_pass http://app_upstream;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_buffering off;
proxy_cache off;
gzip off;
proxy_read_timeout 3600s;
}
If you cannot edit the proxy config — a shared ingress, a managed load balancer — the application can ask for the same behavior with a response header. nginx honors X-Accel-Buffering: no per response, which makes it the one reverse proxy where a hosted app can fix its own streaming without touching infrastructure config.
Set the headers correctly at the origin regardless:
from fastapi.responses import StreamingResponse
import json
@app.get("/api/chat")
async def chat():
async def gen():
async for chunk in llm_stream():
yield f"data: {json.dumps({'text': chunk})}\n\n"
yield "data: [DONE]\n\n"
return StreamingResponse(
gen(),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache, no-transform",
"X-Accel-Buffering": "no",
},
)
no-transform is the part people leave out. It tells intermediaries they may not re-encode the body, which is a standards-level way of asking CDNs not to compress your stream.
The takeaway: fix compression first, proxy buffering second — in that order, because a compression layer will keep buffering even after you turn proxy buffering off.
What if the platform itself refuses to stream?
Some runtimes buffer the whole response by design, and no header will change that. Classic API Gateway integrations return the response as a single payload — the Lambda finishes, then the gateway replies. If you are streaming from AWS, Lambda Function URLs with the response-streaming invoke mode and awslambda.streamifyResponse are the supported path, at the cost of moving off API Gateway and losing the request-level features you had there.
Platform-side, as of September 2026, Vercel, Cloudflare Workers, and Fly.io all support streamed responses on their standard runtimes, so a stream that dies there is usually your own middleware rather than the platform. Cloudflare Workers is the one I reach for when the workload is pure passthrough streaming, though the CPU-time model makes it a poor fit if you do heavy work in the same request. If you need a long-lived connection with ordinary Node or Python semantics and no execution-time ceiling, a container platform like Fly.io removes the constraint entirely — you trade the zero-ops story for managing a process that stays up.
The takeaway: before optimizing your streaming code, confirm the runtime is even allowed to send bytes before the handler returns.
Is the bug possibly in the browser?
Sometimes. If the network panel shows chunks arriving over time but the UI updates once, the server is fine and the client is collecting the body. await res.text() and await res.json() both wait for completion by definition. Read the stream:
const res = await fetch("/api/chat");
const reader = res.body.getReader();
const decoder = new TextDecoder();
while (true) {
const { done, value } = await reader.read();
if (done) break;
render(decoder.decode(value, { stream: true }));
}
Note { stream: true } — without it, a multi-byte UTF-8 character split across two chunks decodes to a replacement character. That one bites when non-English output enters the picture.
FAQ
Why does my SSE stream only work locally and not behind nginx?
nginx has proxy_buffering on by default, so it accumulates the proxied response before forwarding it. Set proxy_buffering off for that location, or have your application send the X-Accel-Buffering: no response header, which nginx honors per response.
Does gzip break server-sent events?
It can. gzip needs input to accumulate before it emits compressed output, so a compression layer holds your tokens even when every proxy in front is configured to stream. Exclude text/event-stream from compression and send Cache-Control: no-cache, no-transform.
How do I test if a response is actually streaming?
Run curl -N --no-buffer against the endpoint and timestamp each line as it arrives. If all lines carry effectively the same timestamp, something buffered the response; if the timestamps spread out, the stream is live. Test the origin port first, then each hop outward.
Bottom line
If tokens stream on localhost and not in production, spend your time bisecting hops rather than rewriting the handler. Check in-app compression first, then reverse-proxy buffering, then the CDN, then the runtime — in that order, since each earlier layer can mask a fix applied to a later one. Set Cache-Control: no-cache, no-transform and X-Accel-Buffering: no at the origin as a permanent default; they cost nothing and preempt the two most common causes. And keep the timestamped curl -N one-liner in your notes — it converts a vague "streaming feels broken" report into a specific hop in about ten minutes.
Top comments (0)