Mustafa ERBAY

Posted on Sep 16 Originally published at mustafaerbay.com.tr

Written but Not Sent: The 487 KB Inside the Kernel

#linux #networking #tcp #kernel

In 2018 Cloudflare published three lines to fix nginx's HTTP/2 prioritization; two were for BBR and fq, the third was this:

net.ipv4.tcp_notsent_lowat = 16384

The rationale was one sentence: once response data has been written into the TCP send buffer it is beyond the server's control and has been committed to be delivered in the order it was written. In HTTP/2, if the server has piled a low-priority image into the queue and a high-priority CSS arrives, the CSS waits behind that image; the bigger the buffer, the longer the wait. When I read the post I liked the setting and did not apply it to my own server; "isn't 16 KB too small, won't throughput drop" went through my head and I left it. Eight years later I measured it. This post is that measurement: how far is it from the kernel accepting a send() call to the byte reaching the wire, who decides that distance, and what you lose by shortening it.

The two halves of the queue

A TCP socket's write queue is not one number but two pieces. In the kernel source their names are snd_una, snd_nxt and write_seq: between snd_una and snd_nxt are bytes sent but not yet acknowledged (data in flight), between snd_nxt and write_seq are bytes the kernel has accepted but never sent. ss -tmi shows the second piece in the notsent: field. When the application calls send() the bytes enter the second piece and the call returns; what the application thinks is "sent" is actually "queued".

How large does the kernel let the unsent piece grow? By default the only limit is the send buffer itself: the third value of tcp_wmem, 4 MiB on my server, and autotuning can grow the buffer up to there according to the bandwidth-delay product. The trouble is that a large buffer serves two different purposes and does not separate them: it needs to be large for data in flight (on long fat pipes cwnd is big), and there is no benefit at all in it being large for unsent data; that piece exists only so the kernel has work in hand while the application sleeps, and a few RTTs' worth of data is enough for that.

tcp_notsent_lowat makes exactly this separation. The commit message Eric Dumazet wrote in 2013, for Linux 3.12, states the intent plainly: the write queue needs to be large only to deal with a large BDP, not to cope with scheduling delays; incoming ACKs make room for the application. For most workloads 128 KB or less is enough for the application to react to POLLOUT in time. With 200 concurrent netperf flows, TCP buffer memory dropped from 45,458 pages to 20,567, 55 percent. The side effect Cloudflare used, latency, was not the commit's first goal; Dumazet wrote it for memory.

Two network namespaces on my server

I did not want to touch the live server's sysctl, and there was no need: net.ipv4.tcp_notsent_lowat is a per-network-namespace value. I created two ip netns, put a veth pair between them, and attached netem with 20 ms delay and a 10 Mbit rate limit to the sender's egress. On the far side a Python process listening on port 9000 and counting everything it reads; on this side another one writing 16 KiB chunks from a poll() loop on a non-blocking socket. At the third second the sender slips a line into the stream: URGENT@<timestamp>#. When the receiver sees that line it prints the difference. This is Cloudflare's "CSS waiting behind".

Three runs, the only difference being the TCP_NOTSENT_LOWAT value given to the socket:

lowat=default     URGENT arrived: delay=0.490s   ss: notsent:486640  unacked:75  rtt:90.4ms
lowat=16384       URGENT arrived: delay=0.093s   ss: notsent:19096   unacked:69  rtt:82.3ms
lowat=131072      URGENT arrived: delay=0.162s   ss: notsent:81840   unacked:83  rtt:97.2ms

With the default, the urgent line waited 490 milliseconds. The ss snapshot says why: notsent:486640, 487 KB the kernel had accepted and not yet sent. On a 10 Mbit link, 487 KB is exactly 0.39 seconds; the rest is the netem queue and the RTT. With a 16 KiB threshold the same line arrived in 93 milliseconds, with 19 KB unsent. At 128 KiB, 162 milliseconds.

On throughput I fooled myself the first time: the totals the sender had written into the kernel in six seconds came out as 7.73, 7.29 and 7.34 MB, and I had noted "the 16 KiB threshold carried 5.7 percent less". But at most 7.5 MB can pass through a 10 Mbit link in six seconds; 7.73 cannot have passed. The difference was data waiting in the kernel queue and draining after the sixth second, not data on the wire (7.73 − 7.29 = 0.44 MB, exactly the notsent difference between the two runs). I measured again on the receiver with a fixed six-second window: 7.16 MB in all three runs, 9.54-9.55 Mbit/s. A 16 KiB threshold was enough to fill this link; on a 10 Mbit, 20 ms link the base BDP is 25 KB, but with netem's 50-packet queue the measured cwnd is around 70 packets, about 100 KB in flight; a 16 KB unsent queue is enough to keep the kernel busy until POLLOUT arrives and the application writes. Dumazet's commit says 128 KB has "no bad effect on the throughput"; on this link the same holds for 16 KB. The risk of running empty is the application's reaction time to POLLOUT times the link speed; the 2022 fix commit describes it as "a fraction of what CWND and pacing rate would allow to send during this RTT".

On the first attempt I had left netem its default 1000-packet queue, and the RTT measured 159 to 220 ms, unacked 131 to 201 packets. That was data waiting not in the socket but in netem's own queue; I was not measuring what I meant to measure. Dropping the queue to 50 packets brought the RTT to 80-97 ms and produced the numbers above. Every buffer on the path has its own delay, and tcp_notsent_lowat shortens only the one in the socket; it can do nothing about bloat in the router.

POLLOUT arrives at the half

Looking at the source turned up a detail I did not expect. The threshold check is one line (in net/ipv4/tcp.c on master, in tcp_ipv4.c on the 6.8 I measured on, same logic):

u32 notsent_bytes = READ_ONCE(tp->write_seq) - READ_ONCE(tp->snd_nxt);
return (notsent_bytes << wake) < tcp_notsent_lowat(tp);

The wake parameter comes in as zero from the sendmsg() path and one from the poll() path. With zero the comparison is plain: if the unsent bytes are below the threshold, write. With one the left side is doubled; that is, poll() waits for the unsent piece to drop below half the threshold before declaring the socket writable. Hysteresis: so the application does not wake up just under the threshold, write a few bytes, and sleep again. The comment above the function in 6.8 says it outright: "This sends EPOLLOUT only if notsent_bytes is half the limit. This mimics the strategy used in sock_def_write_space()."

I measured it. A small loop reading the tcpi_notsent_bytes field of TCP_INFO on every POLLOUT return:

lowat=65536 chunk=4096   polls=1173  notsent@POLLOUT: max=32760  | after send max=36856
lowat=65536 chunk=16384  polls=300   notsent@POLLOUT: max=32744  | after send max=49128
lowat=16384 chunk=16384  polls=297   notsent@POLLOUT: max=8184   | after send max=24568

With a 64 KiB threshold, poll() never said "writable" with a value above 32,768; with a 16 KiB threshold, never above 8,192. On the other hand a single 16 KiB send() pushes the unsent piece to 24.5 KB. tcp_sendmsg_locked checks the threshold not on every byte but when it needs to open a new segment (the new_segment: label); the segment goal is size_goal, up to 64 KB with GSO, and if there is room in the segment at the tail of the queue the new write is appended there without a check. So the threshold is exceeded by at most one segment; a 1 MiB sendfile(), however, stops in the middle, because the question is asked again at every new segment (in the kTLS post I counted nginx's sendfile calls with strace; those calls get split the same way).

The threshold also had a known pit: the kernel generated EPOLLOUT only when an ACK arrived and snd_una advanced. On a flow with a long RTT or receiving SACKs, if the unsent piece drops to zero and snd_nxt advances but snd_una stays put for an RTT, the application was not woken; queue empty, application asleep. Dumazet fixed this in April 2022 with 4bfe744ff164 (5.18, backported to stable): POLLOUT is now also generated when snd_nxt advances. If you see odd stalls with a small threshold on an old kernel, the absence of this commit may be why.

Sysctl or socket option

There are two layers and the precedence is clear: tcp_notsent_lowat() looks at the socket's own value first and falls back to the sysctl if it is zero. The sysctl takes effect immediately, for running sockets too; the ip-sysctl document says so explicitly. I tested the namespace boundary: in the sender namespace I ran sysctl -w net.ipv4.tcp_notsent_lowat=16384 and started the sender with no socket option at all; the urgent line arrived in 124 ms, notsent:11176; the value in the main namespace stayed at 4294967295. You can hand this setting to a containerised service via sysctls: without changing the host.

So who uses it? I searched source repositories for TCP_NOTSENT_LOWAT:

nginx: no. There is a directive called send_lowat, but that is SO_SNDLOWAT, a different thing (Dumazet's commit specifically stresses the difference), and its documentation says it is ignored on Linux. This is why Cloudflare went the sysctl route: there is no knob to give nginx. The 2018 post said "we have a patch we are preparing to upstream for NGINX"; eight years later the nginx source still has zero matches.
H2O: yes, and the most interesting one. lib/common/socket.c sets the threshold dynamically per connection for its HTTP/2 latency optimisation, pushing it down to 1 byte based on RTT and cwnd (the http2-latency-optimization-* directives). What Cloudflare does coarsely with a sysctl, H2O does per socket with fine tuning.
HAProxy: yes. tune.notsent-lowat.client and tune.notsent-lowat.server, applied with setsockopt on every connection; default zero, meaning unlimited. The documentation calls it "more effective and more accurate" than tune.sndbuf.
Go, Node/libuv, curl, Apache httpd, Caddy, Envoy, OpenSSH: their code bases have only the constant definition or nothing at all. In Go the constant sits in golang.org/x/sys/unix; the net package does not use it.

On my own server: the sysctl is at its default; as I wrote this, nginx had 46 established HTTPS sockets and all 46 had a skmem write counter of zero (an hour earlier it was 84, again all zero). The setting would have no measurable effect here, because the server sits behind Cloudflare and HTTP/2 prioritization happens at the edge. Where the setting belongs is an nginx that talks to browsers directly and serves HTTP/2.

Why shrinking the buffer is not enough

The first question that comes to mind: if the problem is a big buffer, why not just shrink SO_SNDBUF? Because it does not work; I tried. Same lab, this time SO_SNDBUF instead of TCP_NOTSENT_LOWAT (the kernel doubles the given value):

SO_SNDBUF=16384 (tb32768)   URGENT delay=0.028s  received in 6 s: 4.64 MB = 6.18 Mbit/s  cwnd:48  unacked:12
SO_SNDBUF=65536 (tb131072)  URGENT delay=0.052s  received in 6 s: 7.16 MB = 9.54 Mbit/s  cwnd:99  unacked:53

The urgent line arrived in 28 ms, the best latency result of all; but the link stayed at 62 percent of 10 Mbit. Read unacked:12 next to cwnd:48: the congestion window allows 48 packets, the socket has no room to hold them, 12 packets are on the wire. Because SO_SNDBUF puts unsent data and in-flight data in the same bucket, it squeezes both. In the tcp_notsent_lowat runs, unacked equalled cwnd every time (75/75, 69/69, 83/83 in the first run; 74/74, 71/71, 70/70 in the second): it had shortened only the waiting part without touching the data in flight. The HAProxy documentation describes this: the socket is reported full once the buffered data reaches "this value plus the measured window size"; it is "more effective and more accurate" than tune.sndbuf because it leaves the window out of the calculation.

The recipe

For an nginx serving HTTP/2 directly to browsers the only route is the sysctl; a file under /etc/sysctl.d/, the value above the BDP and somewhere between Cloudflare's 16 KB and the commit's 128 KB. The change applies immediately to running connections too, no restart needed. In HAProxy, tune.notsent-lowat.client and .server in the global section; the documentation suggests a value around tune.bufsize, with the client and backend directions set separately. In a container, sysctls: net.ipv4.tcp_notsent_lowat=16384; since net.* is per namespace the host does not change. In a server you write yourself, setsockopt(IPPROTO_TCP, TCP_NOTSENT_LOWAT) per connection; leave it at zero if you do not want to override the sysctl, the kernel falls back to it.

Then measure. If notsent in ss -tmi stays above the threshold for long, the application is reacting to POLLOUT late; if unacked stays below cwnd, the problem is not this setting but SO_SNDBUF or the upper bound of tcp_wmem.

What it does not fix

This setting shortens a socket's unsent queue inside the kernel, that is all. Not the application's own buffer: nginx keeps holding the rest of the file it sends with sendfile, Node keeps its own write queue; the threshold only moves forward the moment the kernel says "give". Not the data in flight either: as many packets as cwnd and the peer's window allow are on the wire, that piece is in the unacked column. There is a separate knob limiting bytes queued in the local qdisc and NIC, tcp_limit_output_bytes (TSQ); it is independent of this threshold too. Not the queues in network devices at all; the 220 ms of the first attempt is the proof. The cost to throughput appears if the application answers POLLOUT later than the link speed requires; it did not appear in my loop, but it can in a heavy event loop, and the commit message spells out the price: more sendmsg()/sendfile() calls on non-blocking sockets, more context switches on blocking ones.

When choosing a value I have two references. The commit says 128 KB, Cloudflare 16 KB. Both are right; the first was written for memory, the second for HTTP/2 priority, and Cloudflare's links are far wider than my netem. The yardstick is not the BDP itself but the bytes the link will consume before the application answers a POLLOUT: link speed times event-loop latency; keep the threshold above that and well below the buffer, then look at notsent in ss -tmi (the ss reading habit from the keepalive post serves here too); if the number swings between half the threshold and threshold-plus-one-chunk, the setting is working.

The lab was built with two namespaces and torn down; the server's sysctl did not change.

Measurements on Ubuntu 24.04, kernel 6.8.0-139-generic, iproute2 6.1.0, Python 3.12; netem delay 20ms rate 10mbit limit 50. tcpi_notsent_bytes was read at byte 144 of struct tcp_info and compared against ss output. Cloudflare's 2018 post: Optimizing HTTP/2 prioritization with BBR and tcp_notsent_lowat. Code searches via GitHub, 16 September 2026.

DEV Community