nginx streams your tokens fine. HAProxy holds them for 206ms.

The advice is everywhere: if you are streaming Server-Sent Events through
nginx, turn off proxy_buffering or your tokens will arrive in one lump.

I built a rig to measure it. nginx does not buffer a token stream. The proxy
that does is HAProxy, on a stock config, and it holds your first token for
206 milliseconds. The header people reach for to fix SSE buffering has no
effect on it, because that header is an nginx convention and nginx was never
the problem.

Four proxies, pinned to exact patch versions, all in front of the same
emitter, measured on a Linux droplet:

proxy	time to first token	gap between frames	frames per read
direct (no proxy)	2ms	50.0ms	1.02
nginx 1.31.5	3ms	50.0ms	1.02
nginx 1.31.5, proxy_buffering off	2ms	50.0ms	1.02
Caddy 2.11.4	2ms	50.0ms	1.02
Traefik v3.7.13	2ms	50.0ms	1.02
HAProxy 3.4.4	206ms	0.0ms	5.12

The emitter sends a frame every 50ms. Through nginx, Caddy and Traefik the
frames arrive 50ms apart, one per read, indistinguishable from no proxy at
all. Through HAProxy they arrive in bursts of five with zero gap between
them, 206ms late.

proxy_buffering is on by default in that nginx row. I checked with
nginx -T inside the running container rather than trusting the config file.
Turning it off changes nothing.

The metric

Everything here rests on one number: frames per read. Take the number of
SSE frames the client received, divide by the number of recv() calls that
delivered at least one. 1.0 means every frame arrived on its own. 41.0 means
the entire stream landed in a single read.

It is a ratio, so it needs no baseline to interpret and it survives a noisy
host in a way that absolute latency does not.

Before trusting it I pointed it at a relay that deliberately drains an entire
upstream response and then flushes it in one go. Direct: 1.00. Through the
relay: 21.00. The instrument detects coalescing when coalescing is there.

Two numbers in this post are not findings, and I want that on the record
before the tables start:

• Frames per read has a noise floor around 1.02. The direct, unproxied path measures 1.02. One merged read in 41 is the client and the kernel, not a proxy. Anyone reporting 1.02 as distinct from 1.00 is reading noise.
• This host produces isolated 30-45ms scheduling spikes with no proxy and no Docker in the path at all. Emitter alone, 40 runs: p90 of 10.89ms, max of 44.50ms. So no ~40ms first-token stall can be pinned on a proxy from this rig. More on that below, because I nearly published one.

Token-sized frames are HAProxy's worst case

This is the part that makes it matter for anyone streaming from a model.

Same 50ms interval, same HAProxy, same stock config. The only change is
growing each frame from about 60 bytes to about 1.1KB:

frame size	time to first token	gap between frames	frames per read
~60 bytes	206ms	0.0ms	5.12
~1.1KB	53ms	41.0ms	1.46

Grow the payload and the coalescing largely goes away. First token four times
faster, frames arriving spaced near the emit interval instead of in bursts,
close to one frame per read.

Varying the emit rate points the same direction. Frames per read goes 13.67 at
a 5ms interval, 5.12 at 50ms, 2.05 at 200ms. Faster emission means more bytes
accumulating per unit time, and more bytes means earlier flushes.

So the trigger is substantially about filling a buffer, not a fixed timer. I
am not going to name a byte threshold: the values implied by different cells
do not reconcile cleanly, around 2.2KB from one and 1.05KB from another, and
pinning it down properly needs a dedicated sweep I have not run.

But the practical consequence is sharp. An LLM token stream is small frames
arriving steadily, which is exactly the shape that triggers this. Benchmark
the same proxy with realistic 1KB chunks and you will not see it. That is
probably why it is not better known.

It also means the frames-per-read figure is a property of this stream through
this proxy, not a property of HAProxy. Quoting "HAProxy delivers 5 events per
read" as a fact about HAProxy would be wrong; it delivers 13.67 per read at a
5ms interval and 2.05 at 200ms.

The header everyone reaches for does nothing

X-Accel-Buffering: no is the documented escape hatch for exactly this
problem. I sent it and measured again:

	time to first token	frames per read	max gap
HAProxy, no header	206ms	5.12	256.4ms
HAProxy, X-Accel-Buffering: no	214ms	5.12	256.2ms

Identical. The 206 against 214ms is run-to-run variation; the ratio and the
max gap match to three significant figures.

It is an nginx convention. nginx honours it, and nginx was not buffering.
HAProxy has never claimed to read it. So the standard fix for SSE buffering
is a header that the only buffering proxy in this set ignores.

When proxy_buffering does cost you something

I could not leave the nginx result as "the advice is unnecessary", because
every cell above uses a client that reads promptly and a stream of about
2.4KB total. nginx's default proxy_buffers is 8 × 4k or 8k, so 32 to 64KB.
The buffers were never close to full. The directive had nothing to do.

So I built the case where it does: about 328KB of payload, pushed through a
client that deliberately sleeps 200ms between reads.

endpoint	time to first token	frames per read
direct	2ms	3.73
nginx (proxy_buffering on)	53ms	3.73
nginx (proxy_buffering off)	3ms	3.57

There it is. Buffering on costs about 50ms of first-token latency, and it is
consistent: median 53ms, max 54ms across ten runs, never once fast.

Note what did not happen. Frames per read is 3.73 either way, the same as
direct. The coalescing in that column is the slow client and the kernel. The
directive cost latency; it did not turn the stream into one lump.

So the honest position on proxy_buffering is neither of the two things the
internet tells you:

• Small frames, prompt client — the actual token-streaming case — it costs nothing measurable, across four separate cells.
• Bulk payloads, slow client — it costs about 50ms of first-token latency.
• In no cell did it cause the dramatic coalescing it is famous for.

The finding I killed

Early on, measuring on macOS with Docker Desktop, I saw Caddy and Traefik
consistently take about 42ms to first token while nginx took 2ms. The first
request after startup was fast and every reused one was slow, which is the
signature of a delayed-ACK stall on a pooled upstream connection. I had a
mechanism, I could predict which condition would reproduce it, and I built a
2×2 to demonstrate it.

Three things then went wrong with that story, in order.

A reviewer pointed out that my nginx config has no upstream{} block and no
keepalive, so nginx never pools upstream connections at all. It was
permanently in the "fresh connection" regime that Caddy and Traefik only hit
on their first request. I had not been comparing proxies. I had been comparing
pooled against unpooled.

Then I ran the emitter on its own, with no proxy and no Docker in the path,
and it produced 30 to 45ms spikes by itself. Whatever I had been measuring, I
could not attribute it to a proxy.

Then I moved to Linux and it vanished entirely. Across all nine cells and
every condition — pooled or fresh, Nagle on or off — Caddy and Traefik sit at
2 to 5ms. The fresh-conn cell restarts the whole stack five times so every
row is a genuine first request: Caddy 5ms, Traefik 4ms. No difference from
pooled.

It was a Docker Desktop measurement artifact. Reporting it as a proxy
behaviour would have been wrong, and I got within one experiment of doing it.

Does it hold against a real model

Every number above comes from a synthetic emitter, because exact pacing is
what makes the comparison causal. So I swapped the upstream for DigitalOcean's
serverless inference endpoint and measured through the same client, 15 calls,
all HTTP 200:

endpoint	frames per read	gap between frames	frames
HAProxy	5.43	0.0ms	169
nginx	1.04	8.6ms	185
nginx, proxy_buffering off	1.02	8.7ms	176

Synthetic said 5.12 against 1.02. Real tokens from gpt-oss-20b say 5.43
against 1.04.

This table cannot prove causation on its own — a model's time to first token
varies between calls, which is why the synthetic cells carry the argument. Its
only job is to show the effect is not an artifact of my emitter. It is not.

How the numbers were kept honest

Two guards gate everything, and both can void a result.

The emitter records its own send timestamps and exposes them, so after every
run the harness checks whether the emitter actually paced as instructed. If it
drifted, that run is discarded rather than blamed on a proxy. The direct
unproxied path also has to show a clean incremental stream, or no proxy row
from that host means anything.

The suite makes exactly one attempt per cell and never retries. That rule
exists because I broke it. An early run "succeeded" on the sixth attempt, and
when I stopped retrying, one honest attempt certified one cell out of six.
Re-running until the guards pass selects for quiet moments on the machine and
biases every number, invisibly.

Individual runs are discarded, at a rate of 1.7% to 11.7% depending on the
cell, because across roughly 2,400 timed writes per cell the chance of one
scheduling hiccup approaches certainty. That exclusion is only defensible
because it is declared in advance, mechanical, measured on the emitter side
independently of the proxy being tested, and every exclusion is counted in the
published table. Re-running until green fails all four of those tests.

The internal check that convinced me it was working: re-enabling Nagle on the
emitter doubled the exclusion rate from 5.0% to 10.0% and left every proxy
number unchanged. Exclusions were removing apparatus noise, not shaping the
result.

Numbers come from a 2-vCPU Ubuntu 24.04 droplet, not a laptop. Docker Desktop
on macOS routes container traffic through a VM network stack, which puts tens
of milliseconds of noise into a measurement whose signal is 50ms — and, as
above, invented a finding.

That is the short version. The harness itself was wrong in fourteen ways
before it produced a number I would stand behind: a guard that audited one run
in ten, a pacing timestamp on the wrong side of a blocking write, a healthcheck
perturbing the thing it was checking, and the retry loop above doing to my own
numbers precisely what the guards existed to prevent. Every one of those, and
the experiment that proved each fix actually bites, is written up separately in
My benchmark harness was wrong fourteen ways before it measured anything.

What I would still test

• The exact flush trigger in HAProxy. A payload-size sweep at a fixed interval would settle whether it is a byte threshold, a timer, or both.
• Whether option http-no-delay removes it, and what that costs.
• Caddy and Traefik with upstream keepalive explicitly configured, since my configs leave pooling at vendor defaults and nginx's default is no pooling at all.

The harness is stdlib Python and Docker, no dependencies. selftest.sh
certifies the host before any measurement and refuses to run against a server
it did not start. run.sh writes the image digests and the full text of every
proxy config into the results file, so the configs in this post cannot drift
from the ones that produced these numbers.

If you are streaming tokens through HAProxy on a stock config, that is where
your first 206 milliseconds went.

Comments

[Raknaos]

The frames-per-read ratio is the part I keep thinking about. Most buffering debugging I've seen anchors on absolute latency deltas, which is exactly the number a noisy shared host destroys — and then you start blaming a proxy for a 40ms scheduler hiccup.

We run a fleet of agents behind one reverse proxy on a VPS and our streams are all small token-sized frames on a steady interval. Never once thought to test whether the proxy was coalescing them; we assumed nginx. What's the emitter you used — is the rig published somewhere? I'd like to point it at our config before I conclude we're fine.