Mustafa ERBAY

Posted on Sep 20 Originally published at mustafaerbay.com.tr

TCP Fast Open: One RTT in the Lab, One Second on the Road

#linux #kernel #tcp #sysctl

A commit that landed in Chromium in December 2018 opens with: "We never enabled it by default, and have no plans to, so we should just remove it." The thing being removed was TCP Fast Open support; the reasons listed were that QUIC does the same job, that it can mutate state alongside 0-RTT session resumption, and that because the cookie is held in global system state it leaks data between profiles, incognito ones included. Firefox followed in version 87, and the Go team closed the "add TFO support" request that had been open since 2013 with a 2020 note saying it was no longer desired. On the kernel side, 2021 brought a change that turned off TFO's blackhole protection by default.

Against that, my own server has net.ipv4.tcp_fastopen = 1. Ubuntu 24.04, kernel 6.8, never set by hand anywhere; that is the kernel default. Over ten days of uptime nstat -az reported TcpExtTCPFastOpenCookieReqd 5: five SYN packets had asked for a cookie and the server had answered none of them. TCPFastOpenActive 0, meaning not a single TFO connection had left this machine in ten days; ip tcp_metrics show holds 48,903 destination entries and not one carries an fo_cookie. The protocol is "on" on every machine, and nobody is sending or accepting anything.

Why the browsers walked away is covered above; what I actually wanted to know is what each knob does in the places where TFO is still alive: curl, nginx, HAProxy, DNS-over-TCP resolvers, and clients you write yourself. So I built a lab out of two network namespaces on the VPS, stretched the link between them to an 80 ms RTT with netem, and tested every claim against a packet capture, from the sysctl bits to queue overflow, from key rotation to blackhole behaviour. The two numbers in the title come from there: a GET that fits in the SYN saves exactly one RTT in the lab, while a box that drops the SYN adds exactly one second to every connection, and no TFO counter shows it. Because the tcp_fastopen sysctl is per network namespace (sock_net(sk)->ipv4.sysctl_tcp_fastopen), I never touched the server's own setting; the root namespace stayed at 1 throughout.

The protocol in two sentences, the cookie in one

The idea in RFC 7413 is simple: on the first connection the client puts an empty Fast Open option in the SYN to request a cookie, and the server hands back a cookie of 4 to 16 bytes in the SYN-ACK. On later connections the client puts the cookie and application data in the same SYN; if the cookie checks out, the server delivers the data to the application before the three-way handshake finishes, and a full RTT is saved.

The cookie is a note the server writes to itself. On Linux it is 8 bytes, generated in tcp_fastopen.c as a SipHash over the source and destination IP addresses; the port plays no part. I saw that directly in the lab: every SYN to ports 80, 81 and 9000 on 10.77.0.1 carried the same cookie, 59b0725f4ceafa2b. The key is one tcp_fastopen_key per network namespace (generated randomly on first use), and the cookie is cached on the client in tcp_metrics, per destination IP.

Section 2.1 of the RFC is candid about what TFO gives up: data in a SYN can be delivered twice, either through a stale duplicate SYN arriving after the server has rebooted, or through a duplicate arriving after the connection was closed. That is why section 6.1 says a server that cannot accept receiving the same SYN data more than once MUST NOT enable TFO; the nginx documentation for fastopen= repeats the sentence almost word for word. Fine for an HTTP GET, a problem for a POST without a transaction identifier.

The lab: one GET request over an 80 ms link

In the server namespace nginx-debug listens on three ports: a plain listen on 80, listen ... fastopen=256 on 81, fastopen=1 on 82. The server side has tcp_fastopen=3 (client and server bits); the client side keeps the default 1. The client is curl --tcp-fastopen; on Linux libcurl implements it with the TCP_FASTOPEN_CONNECT socket option, so connect() sends no SYN and the SYN goes out with the first write().

The first four requests to port 81, time_connect / time_starttransfer in seconds:

0.080853 0.161949   ← cookie request: empty SYN, SYN-ACK carries the cookie, 2 RTT
0.000159 0.081023   ← 75-byte GET inside the SYN, response after 1 RTT
0.000185 0.081972
0.000182 0.081070

In the capture the first SYN shows tfo cookiereq and the SYN-ACK tfo cookie 59b0725f4ceafa2b. The second connection's SYN carries the 75-byte HTTP request as seq 740412315:740412390, length 75; the server's SYN-ACK says ack 740412391, acknowledging the SYN plus all 75 bytes of data in one go. Time to first byte dropped from 162 ms to 81 ms, exactly one RTT. The 0.16 ms time_connect is a side effect of TCP_FASTOPEN_CONNECT: from curl's point of view the connection is "established" instantly, because the SYN has not even been sent yet.

The trace left on the client:

$ ip netns exec tfoc ip tcp_metrics show
10.77.0.1 age 1.003sec cwnd 10 rtt 82877us rttvar 31900us fo_mss 1460 fo_cookie 59b0725f4ceafa2b source 10.77.0.2

A small aside: of the three cookie requests sent to port 80 (no TFO there), the second appeared in the capture as exp-tfo cookiereq. When the kernel fails to obtain a cookie from a destination, its next attempt uses the pre-RFC experimental option format (kind 254, magic number 0xF989), and if that fails too it returns to kind 34. The try_exp logic inside tcp_rcv_fastopen_synack() in tcp_input.c exists for this; a leftover from the pre-RFC experimental period.

Server side: 0x2 is not enough, and 0x400 is only checked at listen() time

For a server to accept TFO two things are needed: the 0x2 bit in the sysctl and the TCP_FASTOPEN option on the listening socket. If the application does not set it, the 0x400 bit says "enable it for every listener". I turned 0x400 on underneath a running nginx and sent a data-carrying SYN to port 80:

0.000139 0.163477   ← data in the SYN, but the response comes 2 RTT later

The server ignored the data, the SYN-ACK acknowledged only the SYN, the client resent the data once the handshake completed; TCPFastOpenActiveFail went up by three. The reason is inet_listen() in af_inet.c: the 0x400 bit is consulted only while a socket transitions into LISTEN via the listen() call, and fastopen_queue_tune(sk, backlog) runs at that moment. Changing the sysctl afterwards does nothing for existing listeners; nginx has to be restarted. The same function clamps the queue limit to min(backlog, somaxconn), so when you write fastopen=256, stay below somaxconn.

The fate of the five cookie requests on my server can be read from the same code. The first thing tcp_try_fastopen() does on seeing an empty option is increment TCPFastOpenCookieReqd; whether the server bit is on and whether the listener supports it are checked afterwards. So the counter says nothing beyond "a SYN asking for TFO arrived"; because the server stayed at 0x1 the SYN-ACK goes out without the option, the client gets no cookie, nobody is harmed. I produced the sixth request myself: from the Mac I sent an HTTP request with curl --tcp-fastopen to the server's SSH port (the sshd listener naturally accepted no data) and the counter went from 5 to 6. macOS 26 ships on this machine with net.inet.tcp.fastopen: 3; on Darwin curl uses connectx() with the CONNECT_DATA_IDEMPOTENT flag. To see who the other five were, I recorded every SYN arriving on ports 80, 443 and SSH for 50 minutes: 1,265 SYNs from 329 sources, 418 of them connections opened towards the origin from 278 Cloudflare edge addresses, 847 from direct scanners and my own Mac. The only SYN carrying a Fast Open option was mine. In that window nobody asked for a cookie, Cloudflare's origin connections included; five requests spread over ten days is noise.

The flow below summarises what a 6.8 server does with a SYN, in tcp_try_fastopen() order:

What the queue counts: not the un-accepted, but the handshakes not yet finished

I used to read the number in fastopen=256 as "TFO connections the application has not accept()ed yet"; the measurement showed something else. I wrote a Python server that listens with TCP_FASTOPEN=1 and never calls accept(), then sent one cookie request followed by three SYN+data in a row. All three were accepted; TCPFastOpenPassive rose by three and ss -ltn showed 4 in the listener's Recv-Q. No overflow despite a queue limit of 1, because the client completes each handshake within 80 ms and fastopenq.qlen drops as soon as the final ACK arrives.

I only saw the overflow once I left handshakes unfinished. In the client namespace I added an iptables rule dropping every outgoing packet except SYNs and sent four SYN+data:

State    Recv-Q Send-Q Local Address:Port Peer Address:Port
SYN-RECV 0      0          10.77.0.1:9000    10.77.0.2:44676
SYN-RECV 0      0          10.77.0.1:9000    10.77.0.2:44680
SYN-RECV 7      0          10.77.0.1:9000    10.77.0.2:44666
SYN-RECV 0      0          10.77.0.1:9000    10.77.0.2:44682

TcpExtTCPFastOpenPassive        +1
TcpExtTCPFastOpenListenOverflow  3

The first connection (44666) is in SYN-RECV, yet it holds 7 bytes in its receive queue and already sits in the listener's accept queue; the application could have read it before the handshake finished. The next three fell into overflow and were downgraded to plain connections. The detail that stands out in the capture is that the three overflowed SYN-ACKs carry no Fast Open option at all. The client cannot distinguish that from "this server does not support TFO"; the "XXX" comment above tcp_fastopen_queue_check() admits as much. Section 5.1 of RFC 7413 explains why the limit exists: a spoofed-source SYN flood with valid cookies, unlike a plain SYN flood, actually runs the application and burns CPU and memory. Once the limit is exceeded TFO shuts off and the classic defences such as SYN cookies take over. For the same reason reqsk_fastopen_remove() keeps a TFO request that was cut short by an RST counted in the queue for another 60 seconds; RSTs coming back from the victims of spoofed SYNs fill the queue rather than draining it.

Key rotation: with a backup key you do not lose an RTT

tcp_fastopen_key accepts two comma-separated keys: the primary both generates and validates, the backup only validates. I set the lab key to K2,K1 (K1 the old primary, now backup) and connected with a client holding a K1 cookie:

0.000121 0.081946 200          ← data accepted, 1 RTT
TcpExtTCPFastOpenPassiveAltKey  1
fo_cookie abc8cb6807ee237d     ← the SYN-ACK brought the new cookie

In the no-backup scenario (K1 cookie on the client, K2 alone on the server) the result was TCPFastOpenPassiveFail 1, a SYN-ACK acknowledging only the SYN, data sent after the handshake, 162 ms. But the client still received the new cookie and the next connection was back to 1 RTT (81 ms). So rotation "works" without a backup too; the price is exactly one slow connection per client. If several servers sit behind one IP (RFC section 6.3.4) they all have to carry the same key, otherwise cookies bounce from server to server and produce a steady stream of PassiveFail. An application can also supply a per-socket key with the TCP_FASTOPEN_KEY socket option, in which case the sysctl key is ignored.

The blackhole: the kernel no longer remembers anything

This is the part that surprised me most. RFC section 7.1 reports that roughly six percent of the probed paths drop SYNs carrying data or unknown options, and recommends that implementations negatively cache such incidents. Linux history ran the other way. Until 2017 there was a per-destination syn_loss counter; commit 7268586baa53 from December 2017, citing Microsoft and Mozilla measurements that the problem mostly came from boxes close to the client, turned that into a global pause (an hour, then two, four...). Commit 213ad73d0607 from July 2021 then set the tcp_fastopen_blackhole_timeout_sec default to 0, after complaints that the global logic was far too aggressive. The result on 6.8 today: no memory per destination and none globally. The fo_syn_drops field is still written into tcp_metrics, but the only line of code reading it is the netlink dump.

I measured three middlebox scenarios. First, a rule on the server side dropping SYNs that carry data (--syn -m length --length 100:; a SYN with a cookie but no data is 72 bytes, a data SYN 147):

0.165800 0.248516 200   ← cookie request, no data, passed the box
0.000135 1.205105 200   ← SYN+data dropped, 1 s RTO, plain SYN
0.000250 1.177071 200
fo_syn_drops 2/0.001usec ago fo_cookie 59b0725f4ceafa2b

The SYN+data vanished, the 1-second initial RTO expired, the kernel sent a plain SYN with no option (the comment at the end of tcp_send_syn_data(): "Exclude Fast Open option for SYN retries"), the handshake completed, the data went through. Every data connection: 1.2 seconds. The second one paid the same price while tcp_metrics already said fo_syn_drops 1; the record is kept, not read. I set tcp_fastopen_blackhole_timeout_sec=30 and tried three more times: still 1.2 seconds, despite fo_syn_drops 2, TCPFastOpenActiveFail climbed to 11, TCPFastOpenBlackhole stayed at 0. tcp_fastopen_active_detect_blackhole() fires only on the third consecutive RTO or when the connection times out entirely; because the plain SYN after the first RTO succeeds, it never counts as a blackhole. Even with the protection enabled, this pattern is invisible to it.

The second scenario is worse: let the box drop every SYN carrying option 34, and let the client start from scratch:

1.104646 1.186658 200
1.143795 1.224717 200
1.119631 1.200606 200

Every connection pays a full second on top of the plain handshake. Because a cookie request carries no data, ActiveFail does not move (it stayed at 20 across the three connections) and fo_syn_drops is not written; the only trace is TcpExtTCPSynRetrans, which went from 8 to 11. If you enable TCP_FASTOPEN_CONNECT in your own client and the path is broken, no TFO counter will tell you that you have added a second to every connection. That was the browsers' reason for giving up, and it applies just as much to a client that is not a browser.

The third scenario is cookieless mode. 0x4 on the client (send data without a cookie), 0x200 on the server (accept data without a cookie): 75 bytes of data in the SYN, no option, accepted by the server, 1 RTT. The reflection attack described in RFC section 2.2 targets exactly this mode; it is useful inside a closed network where you know the source IP cannot be spoofed, not on a port facing the internet.

Who uses it, who should

On my server ss -ltnp counts 133 nginx, 35 docker-proxy, five node, three sshd, two HAProxy 3.2 and two Go listening sockets; ten days, zero TCPFastOpenActive, zero TCPFastOpenPassive. The HAProxy containers' configuration has no tfo, nginx has no fastopen. The 0x1 default turns nothing on by itself; the application has to say MSG_FASTOPEN or TCP_FASTOPEN_CONNECT, and very few do. The Go standard library never does; the word FASTOPEN does not appear in the libuv or OpenSSH source trees. The table resembles the one in the keepalive post, with one difference: there, an application that read the sysctl got the right result; here, even an application that asks pays the price on a broken path and never hears about it. The places where TFO still lives are clear from that table: curl (--tcp-fastopen, libcurl CURLOPT_TCP_FASTOPEN), nginx (listen ... fastopen=N), HAProxy (bind ... tfo and backend server ... tfo, whose documentation tells you not to enable it without retry-on conn-failure empty-response response-timeout), and DNS-over-TCP resolvers such as PowerDNS; the complaint cited by the 2021 kernel commit comes from a PowerDNS developer.

Four more production details, all about where the cookie is made. First, both the sysctl and the key belong to the network namespace: enabling 0x2 or 0x400 on the host does nothing for an nginx or HAProxy container on a bridge network, and each container generates its own random key in its own namespace; in Kubernetes net.ipv4.tcp_fastopen is namespaced yet absent from kubelet's safe sysctl list, so it needs --allowed-unsafe-sysctls. Second, unless the key is written to sysctl.d it is regenerated on every reboot, and every client that knows the server pays one full slow connection with a PassiveFail on its next attempt. Third, because the cookie derives from the source and destination address alone, everyone behind the same NAT gets the same cookie (RFC 5.1.1 discusses mixing in the timestamp for exactly this reason and rejects it), and a client behind a CGN whose every connection leaves from a different public IP never gains anything (RFC 7.1). Fourth, once SYN cookies kick in under a SYN flood, tcp_conn_request() does not even parse the Fast Open option; at the moment it would be needed most, TFO is already off.

The questions for your own setup are these. Can the first message be replayed (a GET, a DNS query, an idempotent RPC)? If not, you are done. Do you control the path (same data centre, between your own load balancer and its backends)? If not, do not enable it without measuring, and if you do, watch TcpExtTCPSynRetrans; the TFO counters will not warn you. Is there TLS? The RTT that TFO removes is the TCP handshake's, the one TLS 1.3 0-RTT removes is the TLS handshake's; the two can add up (RFC 6.3.2 considers putting the ClientHello in the SYN safe), and what removes both at once is QUIC. In practice the curl documentation notes that the TLS session cache does not work with TFO enabled, so before enabling TFO for HTTPS, measure which RTT you actually gain. And if you enable fastopen= on the server side, keep the number below somaxconn, never use 0x400, watch ListenOverflow and PassiveFail in nstat, and leave the backup key in place for a while when rotating.

Nothing changed in the root namespace: tcp_fastopen is 1, the cookie-request counter reads 6, and there is still not a single fo_cookie in tcp_metrics. The lab namespaces, the netem queues, nginx-debug and the Python listener have been removed.

TFO really does hand back one RTT in the lab; what was lost is the memory of who pays when there is a box in the middle. The RFC left that to the client, the kernel turned per-destination memory into a global pause in 2017 and switched that off too in 2021, and the browsers moved to QUIC. What you are left with is a switch you can flip between two machines whose path you know; the "1" in every distribution does not mean that switch is on, only that an application that asks may ask.

DEV Community