Your P99 Latency Has a Suspicious 40ms Floor. Two 1980s Algorithms Are Arguing.

The problem

A service I worked on had a weird latency signature: P50 was fine, ~4ms. P99 wasn't a long tail, it was a plateau — a large cluster of requests sitting at almost exactly 40ms, then a normal tail above that. Not "sometimes slow." A specific number, showing up over and over, like the latency had a floor.

Nothing in the app logs explained it. No GC pauses, no lock contention, no slow query. The handler itself, timed in isolation, ran in under a millisecond. The 40ms was appearing somewhere in the network path, and it was suspiciously exact for something people kept shrugging off as "random."

It wasn't random. It was two TCP behaviors from the 1980s, both individually reasonable, quietly canceling out each other's assumptions.

Why it happens

The client was writing a request in two pieces: a small header write, then a separate write for the body a few milliseconds later — an artifact of how the client library serialized the message, header framing then payload. Nothing exotic; plenty of RPC and HTTP client code does this without thinking about it.

Nagle's algorithm (RFC 896, 1984) says: if you have unacknowledged data in flight, don't send another small segment — buffer it, and wait for either an ACK or a full MSS worth of data. It exists so an interactive session (originally: someone typing over Telnet) doesn't flood the network with one-byte packets. Reasonable, on its own.

Delayed ACK (from RFC 1122) says: a receiver doesn't have to acknowledge a segment immediately. It can hold off briefly to see if it'll have outgoing data to piggyback the ACK on, or to combine acknowledgment of multiple segments into one. Also reasonable, on its own — fewer bare ACK packets on the wire.

Put them together and you get a standoff. The client's second write (the body) gets held by Nagle, because the header segment is still unacknowledged. The server has nothing to say yet — it's still waiting for the body before it can respond — so it just delays the ACK, per policy, waiting to see if it'll have something to piggyback on. Both sides are behaving correctly by their own rules. The ACK only goes out when the server's delayed-ACK timer fires, which on Linux defaults to around 40ms (other stacks have historically used up to 200ms). The moment that ACK lands, Nagle releases the body segment, and the request finally completes.

Neither side logs an error. Nothing retries. Throughput graphs look completely normal, because this doesn't cost bandwidth — it costs latency, on a subset of requests, in a way that's invisible unless you're specifically looking at the tail or capturing packets.

What to do about it

The standard fix is TCP_NODELAY — disable Nagle's algorithm on the socket, so small writes go out immediately instead of waiting on an ACK. Almost every mainstream production HTTP client and server sets this by default now, which is exactly why this bug tends to hide in custom protocols, internal RPC layers, or "just open a raw socket" code that never inherited that default.

But TCP_NODELAY alone just trades the problem for a different one (a burst of tiny packets instead of one clean one). The more durable fix is to not create the standoff in the first place: coalesce the header and body into a single write (or a single writev) so there's only one segment in flight, with nothing for Nagle to hold back. That fixes it regardless of which side's delayed-ACK settings you don't control — useful when the other end of the connection isn't yours to configure.

If you suspect this is happening to you, the fastest confirmation is a packet capture on one affected request. You're looking for a small segment sent, silence, then an ACK arriving right before the next segment goes out — with the silence landing close to your OS's delayed-ACK timer. Application logs will never show you this; it lives entirely below the layer your code can see.

Key takeaways

• A latency plateau at a suspiciously round number (40ms, 200ms) is a strong signal of a fixed timer somewhere below your application — delayed ACK is a common culprit.
• Nagle's algorithm and delayed ACK are each sensible in isolation. The bug only exists in the interaction between a sender that splits small writes and a receiver that delays ACKs.
• TCP_NODELAY treats the symptom. Coalescing writes so a request is never split into two small segments treats the cause.
• This is invisible to bandwidth/throughput monitoring and invisible to application-level logging — you need a packet capture to actually see it.