We Cut API Gateway Connections 6x With HTTP/2. One Bad Packet Then Stalled Every Request Sharing It.

The problem

A team I was helping migrated their API gateway's upstream connections from HTTP/1.1 to HTTP/2. The pitch was straightforward: instead of maintaining 6 parallel TCP connections per backend host (the typical HTTP/1.1 client default), multiplex everything over a single connection. Fewer connections, less TCP slow-start overhead, less TLS handshake cost, lower idle memory footprint on the backend. Benchmarks under clean conditions backed it up — p50 dropped, connection count dropped, everyone was happy.

Then a routine network blip hit — the kind that happens between availability zones a few times a month, briefly pushing packet loss to somewhere around 1-2%. Historically this cost the gateway a small, proportional hit: a couple percent of requests got slow or retried. This time, the entire gateway's tail latency spiked. Not 2% of requests — nearly all in-flight requests on the affected hosts.

Why it happens

HTTP/2's multiplexing does exactly what it promises at the HTTP layer: multiple request/response "streams" get interleaved as frames over one TCP connection, so one slow response no longer blocks the next one from starting — the classic HTTP/1.1 head-of-line blocking problem. That part genuinely works.

The problem is one layer down. TCP guarantees in-order, reliable delivery of bytes on a connection. If a single segment is lost, TCP will not hand any later bytes to the application — including bytes belonging to completely unrelated HTTP/2 streams — until the lost segment is retransmitted and the gap is filled. One dropped packet freezes the entire connection's delivery, regardless of how many logically independent streams are riding on it.

With 6 separate HTTP/1.1 connections, a lost packet on one connection only stalls the requests using that one connection — roughly 1/6 of in-flight traffic to that host. Collapse those into a single HTTP/2 connection carrying, say, 40 concurrent streams, and the same lost packet now stalls all 40. You didn't just move the head-of-line blocking problem from HTTP to TCP — you concentrated its blast radius. Fewer connections means fewer independent failure domains.

This is precisely the motivation behind HTTP/3 and QUIC: QUIC runs over UDP and implements its own per-stream loss recovery, so a lost packet affecting one stream doesn't stall the others multiplexed alongside it. HTTP/2-over-TCP structurally cannot do this, no matter how it's tuned.

What to do about it

A few things actually move the needle, in rough order of effort:

• Don't collapse to exactly one connection. Most HTTP/2 client and proxy configs let you cap concurrent streams per connection and open a small number of connections per host (2-4) instead of 1. This costs back some of the overhead savings but bounds the blast radius of a single loss event — it's a direct trade of connection overhead against blocking risk.
• Measure loss, not just latency, on the paths that matter. Most teams monitor p50/p99 and CPU, and few monitor per-path packet loss. If you'd graphed loss on the AZ-to-AZ path already, this incident would have been a two-minute diagnosis instead of a multi-hour one.
• Consider QUIC/HTTP/3 for genuinely loss-prone paths — mobile-facing edges especially, where 1-3% loss is closer to normal than exceptional. It solves this at the transport layer instead of asking you to hand-tune connection counts.
• Don't assume "fewer connections is strictly better." It's a real trade-off. Optimize for it deliberately instead of taking the default multiplexing pitch at face value.

Key takeaways

• HTTP/2 solves head-of-line blocking at the HTTP layer, not the TCP layer — TCP's in-order delivery guarantee reintroduces it underneath.
• Multiplexing more streams onto fewer connections doesn't just save overhead — it concentrates the blast radius of any single packet loss event.
• Under clean-network benchmarks this never shows up. It only bites at the loss rates real production paths hit occasionally, so test — or at least monitor — under loss, not just load.
• QUIC/HTTP/3 exists specifically to fix this, with per-stream loss recovery over UDP.