On my laptop, the chat app streamed like it should. The first word showed up in under a second and the rest typed itself out. Then I deployed it behind nginx, and the same app sat on a blinking cursor for three, five, sometimes nine seconds before dumping a whole paragraph at once.
Nothing was wrong with Claude, and nothing was wrong with my FastAPI code. The cause was nginx proxy_buffering, which is on by default and quietly turns a server-sent events (SSE) stream into a batch job. Fixing it took one line. Getting that one line to actually apply took me most of an evening, because of a second nginx rule I had never thought about.
TL;DR
- nginx has
proxy_buffering onby default. It collects upstream bytes in buffers and writes them to the client in chunks, so an SSE token stream arrives in lumps instead of token by token. - In my replay of 200 prompts, time to first token went from 0.71s p50 (direct to uvicorn) to 3.4s p50 / 9.6s p95 behind nginx. 58 of 200 replies arrived as one single burst.
- The fix is
proxy_buffering off;on the streaming route, or anX-Accel-Buffering: noresponse header from the app. - My first fix did nothing because a regex
locationbeat my prefixlocation. In nginx, regex matches win over plain prefix matches unless you use^~or=. - After the real fix: 0.76s p50 / 1.41s p95, zero single-burst replies. nginx now costs me about 50ms.
What was the setup?
A small internal chat tool. FastAPI app, an endpoint that calls the Anthropic API with stream=True, and a generator that re-emits each text delta as an SSE event. uvicorn runs on 127.0.0.1:8000. In production, nginx terminates TLS and proxies /api/ to uvicorn.
The streaming endpoint looked roughly like this:
@app.post("/api/chat/stream")
async def chat_stream(req: ChatRequest):
async def events():
async with client.messages.stream(
model=MODEL, max_tokens=1024, messages=req.messages
) as stream:
async for text in stream.text_stream:
yield f"data: {json.dumps({'t': text})}\n\n"
yield "data: [DONE]\n\n"
return StreamingResponse(events(), media_type="text/event-stream")
And the nginx side was the kind of config everyone has:
location /api/ {
proxy_pass http://127.0.0.1:8000;
proxy_set_header Host $host;
}
In dev I hit port 8000 directly, so nginx never got a vote. That is why it worked on my machine.
How did I measure time to first token?
I timed the first parsed SSE event on the client, not the first byte of the HTTP response. Headers can arrive early while the body is still stuck, so curl's time_starttransfer told me almost nothing useful here.
I wrote a replay script: 200 prompts from my own chat history, sent one at a time, recording three things per request:
- Time until the first
data:event with text in it (TTFT) - Total time until
[DONE] - How many separate network reads the body arrived in
The third number turned out to be the most telling. A healthy stream arrives in dozens of reads. A buffered one arrives in a handful, or one.
Why does nginx proxy_buffering break SSE streaming?
nginx proxy_buffering breaks SSE because nginx reads the upstream response into memory buffers and forwards it to the client in buffer-sized chunks, not per event. A token stream is hundreds of tiny writes, and nginx is designed to coalesce exactly that.
This is a good default for normal web traffic. Buffering lets nginx slurp a response from a slow app server quickly, free that worker, and then trickle bytes out to a slow client on its own time. For a page of HTML that is pure win. For a stream where the whole point is "show the first word now", it is the worst possible behavior.
Here is what the 200-request replay looked like:
| Path | TTFT p50 | TTFT p95 | Single-burst replies |
|---|---|---|---|
| Direct to uvicorn | 0.71s | 1.32s | 0 / 200 |
| nginx, default config | 3.4s | 9.6s | 58 / 200 |
The 58 single-burst replies were the short answers. Their whole SSE body fit inside nginx's buffers, so the user saw nothing until Claude was completely done, then got everything at once. Longer answers came through in four or five lumps, which looks like a stuttering connection rather than a config problem. That stutter is why I spent the first half hour blaming my Wi-Fi.
How do you disable nginx proxy_buffering for one route?
Add proxy_buffering off; to the location that serves the stream, or send X-Accel-Buffering: no from the app. Either one tells nginx to forward bytes as soon as they arrive from upstream.
So I did the obvious thing and added a dedicated block:
location /api/chat/stream {
proxy_pass http://127.0.0.1:8000;
proxy_buffering off;
proxy_cache off;
gzip off;
}
gzip off is there because compressing a token stream saves almost nothing and is one more filter between the app and the browser. proxy_cache off is there because caching a chat stream makes no sense.
nginx -t, reload, rerun the replay.
TTFT p50: 3.4s. Exactly the same.
Why didn't my nginx location block apply?
My new block never matched, because nginx prefers a matching regex location over any plain prefix location, even a longer and more specific one. Further up the file, sitting there for months, was this:
location ~* ^/api/ {
add_header Access-Control-Allow-Origin $cors_origin always;
proxy_pass http://127.0.0.1:8000;
}
nginx location matching goes like this. It finds the longest matching prefix location and remembers it. Then it checks regex locations in the order they appear. If any regex matches, the regex wins. The remembered prefix is only used if no regex matches.
So /api/chat/stream matched my shiny new prefix block, nginx noted it, then hit ~* ^/api/, which also matched, and used that one instead. Buffering stayed on. For about 40 minutes I was convinced nginx -s reload was broken.
The way I finally proved it was dumb and effective. I added a debug header to each block:
add_header X-Route "stream-block" always;
Then curl -sI against the endpoint. No X-Route header. The request was going somewhere else. nginx -T | grep -n "location" showed me every location in the merged config, including the regex one in an included file I had forgotten existed.
What is the reliable fix?
Use the ^~ modifier so the prefix block wins outright, and also send X-Accel-Buffering: no from the app so the setting travels with the response no matter which block matches.
The ^~ modifier means "if this is the longest matching prefix, stop and do not check regexes":
location ^~ /api/chat/stream {
proxy_pass http://127.0.0.1:8000;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_buffering off;
proxy_cache off;
gzip off;
add_header Access-Control-Allow-Origin $cors_origin always;
}
Note that I had to copy the CORS header in. Once the regex block stops matching, you lose everything it was doing for that path. Check what you are opting out of.
Then, belt and suspenders, in the app:
return StreamingResponse(
events(),
media_type="text/event-stream",
headers={"X-Accel-Buffering": "no", "Cache-Control": "no-cache"},
)
nginx reads X-Accel-Buffering from the upstream response and disables buffering for that response, unless you have told it to ignore the header with proxy_ignore_headers. I like this one because it lives next to the code that streams. If someone later rewrites the nginx config, adds a new regex block, or moves the app behind a different nginx, the streaming route keeps working.
Did it actually fix the numbers?
Yes. Same 200 prompts, third run:
| Path | TTFT p50 | TTFT p95 | Single-burst replies |
|---|---|---|---|
| Direct to uvicorn | 0.71s | 1.32s | 0 / 200 |
| nginx, default config | 3.4s | 9.6s | 58 / 200 |
| nginx, prefix block (shadowed) | 3.4s | 9.5s | 57 / 200 |
nginx, ^~ block + header |
0.76s | 1.41s | 0 / 200 |
nginx now adds roughly 50ms at the median, which is TLS and one extra hop. Total completion time barely moved in any run, by the way. Buffering never made the answers slower to finish. It only made them feel slower, which for a chat UI is the only thing that matters.
The whole investigation cost about three hours and under $2 of API calls for 600 replayed requests.
What would I check first next time?
If an LLM stream works locally and lumps up in production, check these in order:
- Is anything between you and the app buffering? nginx, a load balancer, a CDN. Run the request against the app port directly, then through each hop, and count body reads.
-
Which nginx location actually matched? Add a unique
add_header X-Routeper block and look. Runnginx -Tto see included files you forgot about. -
Is there a regex location that overlaps? If yes, use
^~or=on the streaming route. -
Does the app send
X-Accel-Buffering: no? It costs one line and survives config rewrites. - Measure first event, not first byte. Headers lie.
So why did nginx proxy_buffering break my Claude SSE stream?
nginx proxy_buffering is on by default, so nginx collected my Claude SSE events in memory and forwarded them in chunks, which pushed time to first token from 0.71s to 3.4s and made 58 of 200 short replies arrive in a single burst. Turning it off fixes it, but only if the location you edit actually matches: a regex location ~* ^/api/ silently overrode my prefix block, because regex locations beat prefix locations unless you use ^~. The durable fix is location ^~ /your/stream/path with proxy_buffering off, plus an X-Accel-Buffering: no header from the app so the setting follows the response wherever it goes.
Written by the developer behind Preterview, an interview prep platform.
Top comments (1)
The measurement choice here is the part I would keep: timing the first parsed SSE event rather than the first byte, because headers arriving early is exactly what makes a buffered stream look healthy in a naive benchmark. The other number that matters is the one you almost bury: total completion time barely moved, so this was never a throughput problem, only a perceived-latency one. One check I would add before calling the fix durable is to repeat the same replay through each hop in front of nginx, not just behind it — if a load balancer or CDN re-buffers the stream, the header from the app will not travel with it, and the same single-burst symptom will come back with a different culprit.