Mustafa ERBAY

Posted on Sep 14 Originally published at mustafaerbay.com.tr

tcp_ecn = 2: The Door Is Open, Nobody Knocks

#tcp #ecn #linux #network

The TCP header has carried two flags since 2001: ECE and CWR. When both are lit in a SYN, the client is saying "tell me about congestion by marking packets, not by dropping them." The networking world calls it ECN. The kernel's knob for these flags is net.ipv4.tcp_ecn, its value on almost every Linux box is 2, and the documentation explains that in a single line: accept incoming requests, never ask on outgoing connections.

I had known that for years. What I did not know was who actually knocked on that door. To find out, I listened for nothing but SYN packets on my server's public interface for three minutes (the filter is tcp[tcpflags], so the count is IPv4 only; pcap-filter does not apply that index operator to IPv6):

$ sudo tcpdump -ni eth0 -Q in 'tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0' -w syn.pcap
$ tcpdump -nn -r syn.pcap | grep -o 'Flags \[[^]]*\]' | sort | uniq -c
    517 Flags [S]
     18 Flags [SEW]

535 SYNs, 451 distinct connection attempts once retransmissions are folded, 94 different source addresses. Of the 18 that asked for ECN, 17 came from my own Mac. The remaining one was an address in Turkey. None of the 26 SYNs Cloudflare sent to my origin had the flags set; neither did the scanner bots probing port 5434. The server has been saying "anyone who asks may have it" for twenty-five years, and, not counting my Mac, one person asked in three minutes. To rule out a fluke, while writing this I filtered my Mac out and opened a second, twelve-minute window: 1261 connection attempts, 395 sources, 252 of them Cloudflare; 5 asked for ECN, all from two addresses in Turkey. From Cloudflare, again zero.

This post follows those two flags: who asks, who accepts, how many of the acceptors actually mark, and what is lined up to change this picture in 2026. The measurements come from my own server, my Mac and a Windows Server I manage; for sources I read kernel code; a news headline and a netstat printout had pulled me the wrong way twice, and the code corrected both.

Four doors, four separate decisions

A single "on" is not enough for ECN to work. RFC 3168 describes four separate steps, and at each one the decision belongs to someone else:

The first door belongs to the client: it sets ECE and CWR together in the SYN. The second belongs to the server: it answers with ECE alone in the SYN-ACK. That asymmetry is not an accident; the RFC explains that a single flag was chosen for the SYN-ACK because of broken stacks that reflected the reserved field of incoming headers verbatim. The third door is the sender's again: once negotiation is done, every data-carrying packet gets ECT(0) written into the two-bit ECN field of its IP header. Writing ECT on a SYN or SYN-ACK is forbidden, and pure ACKs must go out Not-ECT, because nothing exists to react to congestion on the ACK path. The fourth door is in nobody's hands: every router in between must preserve those two bits and flip them to CE when congested. The moment one wipes them, the connection is ECN-capable on paper and deaf in practice.

Once the fourth step holds, the receiver sets ECE on its following ACKs the moment it sees CE; the sender shrinks its window as it would for a loss and answers with CWR to say "heard you." The difference from a loss is that nothing is retransmitted. That is the whole gain, and all of it depends on all four doors being open.

What 2 means on Linux, and the rows added to the table in 2026

The table in the kernel documentation now has six rows:

tcp_ecn	Incoming connections	Outgoing connections
0	No ECN	No ECN
1	ECN	ECN
2	ECN	No ECN
3	AccECN	AccECN
4	AccECN	ECN
5	AccECN	No ECN

3, 4 and 5 are new; they arrived with Accurate ECN support in 6.18 (I get to AccECN below). The default is 2, and the first wrong turn was here: last week I had read a headline claiming "AccECN is on by default as of Linux 7.0," and that is how it stuck. I looked at net/ipv4/tcp_ipv4.c; in 6.18, in 7.0 and in today's 7.3-rc3 the same line is there:

net->ipv4.sysctl_tcp_ecn = TCP_ECN_IN_ECN_OUT_NOECN;   /* = 2 */

No Linux release has started asking for ECN on its own. On the server side, what 2 does lives in a few lines of tcp_ecn_create_request() in tcp_input.c: if ECE and CWR arrived together in the SYN and the sysctl is non-zero, accept the request. The same function holds a small trap: if the SYN itself arrived marked ECT, meaning the client broke the rule right at the door, ECN is not granted. On the client side, tcp_ecn_send_syn() (in tcp_output.c on 6.8, in include/net/tcp_ecn.h from 6.18 on) looks for one of three things: is the sysctl at a value that asks for ECN on the outgoing side (only 1 on 6.8; 3 and 4 as well after 6.18), does the congestion control algorithm require ECN (as DCTCP does), or does the route to that destination carry features ecn. If none applies, the SYN goes out plain. On my server none applied.

My Mac asks, netstat says nothing

With 17 of the 18 ECN SYNs coming from the Mac, I looked at the Apple side first. On macOS 26, sysctl net.inet.tcp | grep ecn returns five lines; the two that matter are ecn: 1 and ecn_initiate_out: 1. In the XNU source I found both names wired to the same sysctl_change_ecn_setting handler, and the handler writes both names into a single tcp_ecn variable; right above it sits a TODO saying "remove ecn_initiate_out once the libnetcore ECN cleanup lands." Two names, one switch. When the switch is 1, tcp_set_ecn() enables ECN on the connection. The socket-level TCP_ECN_MODE option can override the sysctl in either direction; beyond that, the main exception is the heuristic in tcp_cache.c: if ECN SYNs have been lost on that network, an RST arrived after the first data, or excessive CE was seen, ECN is not requested for a period that starts at ecn_timeout minutes and doubles.

Then I looked at netstat -s -p tcp, and the second wrong turn was here:

0 client connection attempted to negotiate ECN
0 client connection successfully negotiated ECN

Zero. Zero attempts on a machine that had been up for thirteen days. In the same minute, tcpdump on the server was showing the SYN from that machine like this:

13:36:25.811 IP 5.27.x.x.24126 > 178.18.252.209.52022: Flags [SEW], seq 489129734, win 65535, ...

When a counter and the wire disagree, I believe the wire. And the zero turned out not to be ECN-specific at all: filter the same output through awk '$1+0>0' and a single line survives, the count of open sockets. On macOS 26 this command cannot read any of the TCP counters (the UDP ones are fine); in the XNU source tcps_ecn_client_setup is incremented when the SYN goes out, but the window for looking at it is currently shut. The counter is not lying, it is not speaking. The nicer evidence came in the next few lines. My firewall accepts SYNs to ports 443 and 80 on the main address only from Cloudflare; the SYN from a curl I ran straight from the Mac to 443 was silently dropped, and macOS performed the retreat that RFC 3168 describes as a "MAY" in section 6.1.1.1 to the letter:

13:36:32.911 5.27.x.x.23377 > 178.18.252.209.443: Flags [SEW]
13:36:33.151 5.27.x.x.23377 > 178.18.252.209.443: Flags [S]
13:36:33.366 5.27.x.x.23377 > 178.18.252.209.443: Flags [S]

The first attempt carried ECN; when no answer came, the flags were cleared from the second attempt onward. On Linux the same behaviour is tcp_ecn_fallback = 1, two lines inside tcp_retransmit_skb() annotated with the comment "RFC3168, section 6.1.1.1". That retreat was written in the early 2000s for firewalls that mistook an ECN SYN for a port scan and dropped it, and it still runs today, because some boxes still drop. The price of an ECN SYN is, at worst, one retransmission timeout.

I was curious about Windows too. On a Windows Server 2019 I manage, Get-NetTCPSetting shows EcnCapability: Enabled on all of the Internet, Datacenter, Compat and the two Custom templates. Microsoft's documentation adds a footnote: if the interface-level EcnMarking setting is Disabled or pinned to a specific ECT value, the template setting has no effect. On Windows too there is no single switch; there are two layers, and one can silence the other.

Where ss hides ecn

When I set out to count ECN connections on the server side, my first command was this, and its answer was zero:

$ ss -Htin state established | grep -c ecn
0

Yet my Mac's ssh sessions were open at that moment, and I had just watched them send ECN SYNs. For a while I suspected the kernel; the culprit was ss itself. In the iproute2 source, misc/ss.c prints the ts, usec_ts, sack, ecn, ecnseen and fastopen words only when show_options is set, and that variable is set only by -o, -e or -b. -i alone dumps tcp_info but never writes those words. The right command:

$ ss -Htino state established > ss.txt
$ grep -vc '^\s' ss.txt; grep -c ' ecn ' ss.txt; grep -c ecnseen ss.txt
269
8
7

269 established connections, eight with ECN, seven with ecnseen. The seven are my Mac. The eighth is interesting: a client in Turkey, connected to the nginx on the server's second address, the one not behind Cloudflare, 112 MB uploaded, ecn flag present, ecnseen absent. Its fingerprint is wscale:8 and no timestamps; it looks like Windows, but I cannot name the operating system from the wire with certainty. By the kernel's definition, ecnseen means "we received at least one packet marked ECT from the peer." This client asked for ECN at the door, the server accepted, and it then sent 136 thousand data segments with ECT on none of them. Either its stack negotiates but never marks, or a box in between wipes the bits. I cannot tell which without a tcpdump on that client's side. The result is the same either way: on this connection ECN exists only in the handshake.

That gap between ecn and ecnseen became the axis of the rest of this post.

Negotiating is one thing, marking is another

To make my server ask the question, I did not touch the sysctl; I used ip route's features ecn. One line per destination puts the flags only on SYNs headed there and vanishes when the job is done. The one surprise was that the command insisted on onlink even though my gateway sits inside the same /20; I did not chase the reason through the routing table, not for a /32 that would be deleted when the job was done:

$ sudo ip route add 1.1.1.1/32 via 178.18.240.1 dev eth0 onlink features ecn
$ curl -sk --resolve one.one.one.one:443:1.1.1.1 -o /dev/null https://one.one.one.one/ &
$ ss -tino dst 1.1.1.1
ESTAB ... 178.18.252.209:30220  1.1.1.1:443
     ts sack ecn cubic wscale:13,7 rto:205 rtt:4.295/1.709 ...
$ sudo ip route del 1.1.1.1/32

With the same method I opened HTTPS connections to seven large destinations and looked at the handshake result:

Destination	Address	ECE in SYN-ACK	`ecnseen`
one.one.one.one	1.1.1.1	yes	2 of 6 attempts
github.com	140.82.121.3	yes	no
www.apple.com	23.219.139.53	yes	no
www.microsoft.com	88.221.169.152	yes	no
kernel.org	172.105.4.254	yes	no
dns.google	8.8.8.8	no	—
deb.debian.org	146.75.122.132	no	—

Five accepted, two did not. The right-hand column for the acceptors is what pushed me to write this post. I opened six connections to 1.1.1.1 in total, the last four within the same minute, putting the ss output side by side with the ToS byte tcpdump read from the IP header:

run1 ss:[ecn ecnseen]  tos:[ 2 tos 0x0 ; 18 tos 0x2 ]
run2 ss:[ecn]          tos:[19 tos 0x0 ;  1 tos 0x20]
run3 ss:[ecn]          tos:[20 tos 0x0 ;  1 tos 0x20]
run4 ss:[ecn]          tos:[25 tos 0x20]

tos 0x2 is ECT(0); 0x0 and 0x20 are Not-ECT (in the latter the ECN bits are still zero, only the DSCP field carries CS1). Same anycast address, same minute, four connections: in one the data arrived marked ECT(0), in three it did not, and ecnseen tracked that difference exactly. Of the two earlier attempts, one was marked and one was not; that is where the table's "2 of 6" comes from. My reading is that I landed on different machines behind the anycast address and that those machines, or the paths to them, behave differently; which one it is I cannot say without seeing inside Cloudflare. What I can say is this: ECE in the SYN-ACK is a promise, ECT(0) is the promise being kept, and the two are not the same thing.

To be sure the measurement was not a tool artefact, I set up a control experiment on the Mac with two Alpine containers: client tcp_ecn=1, server tcp_ecn=2, nothing between them but Docker's bridge. On the client side, ss -tino showed ecn ecnseen within the first second. When Linux talks to Linux, all four doors open; when they do not open across the internet, the cause is not at the endpoints but somewhere in between, or in a choice made at the application layer on the far side.

One more note: tc qdisc show dev eth0 on my server says fq_codel ... ecn, and according to the iproute2 documentation ecn is on by default in fq_codel. So my own queue is ready to write CE instead of dropping, should an ECN connection ever hit congestion there. On a VPS's public interface that queue almost never fills; even so, tc -s qdisc show dev eth0 shows ecn_mark 1 in its statistics: one packet out of 70 million was, at some point, genuinely turned into CE. Part of the "should I enable ECN" question has been answered on my behalf for years.

The second attempt, twenty-five years on: L4S and AccECN

Classic ECN has a structural limit: the receiver cannot say how many CE marks it saw within one round trip; the ECE flag stays lit on every ACK until a CWR arrives, and the sender gets a single "something happened" signal. RFC 3168 requires the same reaction to that signal as to a loss: halve the window. That is why classic ECN reduces retransmissions but not latency.

RFC 9330 and RFC 9331 (L4S), published in 2023, propose a different contract: flows that use the ECT(1) codepoint enter a separate, shallow queue at the router, receive CE very early and very often, and in return shrink the window in proportion to the marking rate instead of halving it. For that, the receiver has to know how to count marks; that is what Accurate ECN (AccECN) does, published as RFC 9768 in April 2026. AccECN uses all three flags in the handshake (ECE, CWR and the old NS bit, now called AE), and the flags in the SYN-ACK also feed back the value the SYN's IP-ECN field carried when it reached the server. So the two ends can spot a box that wipes or rewrites the bits during the handshake itself; the "ecn, no ecnseen" connection above is exactly the kind of case that test would catch.

On the Linux side AccECN arrived in 6.18; rows 3, 4 and 5 of the table above, plus the tcp_ecn_option and tcp_ecn_option_beacon sysctls, belong to it. The default is still 2, so AccECN exists only if you turn it on. On the Apple side the plumbing is in place too: my Mac shows net.inet.tcp.accurate_ecn: 0 and net.inet.tcp.l4s: 0, and in the XNU source tcp_set_ecn() also sets up the AccECN state when L4S is enabled. In the autumn of 2026 both major endpoint stacks are ready for the second attempt, and both ship with it off. My three-minute tcpdump contained not a single SYN with the AE bit; the table under this paragraph is worth taking again a year from now.

The decision: where 2 stays, where 1 goes

After these measurements I arrived at the following framework for my own servers; when adapting it to yours I would measure first, because the answer depends on where your traffic comes from.

A web server behind Cloudflare or a similar edge network. The client's ECN ends at the edge; the edge opens the connection to your origin, and in my measurement the edge does not ask. Whatever you write into tcp_ecn here changes nothing for incoming connections. Leave it at 2; instead of changing the setting, seeing once that ss -Htino | grep -c ecnseen returns zero is enough.

A server exposed directly to the internet. Clients do ask; in my case Apple devices and a few addresses in Turkey whose identity I cannot pin down from the wire. 2 is enough, because it accepts when asked. The job here is to count ecnseen, not ecn: if the share of clients that negotiate but never mark is high, ECN is doing nothing for that traffic, and knowing that is worth more than ticking the "ECN on" box.

Connections the server opens itself. Databases, queues, internal services, upstream APIs. Here 2 asks for nothing. If the far side is under your control and the network in between is yours, tcp_ecn = 1 is a low-risk step; with tcp_ecn_fallback = 1 it retreats on its own when a SYN is lost. Know its limit, though: that retreat covers only the SYN. If ECT-marked data falls into a black hole after the SYN got through, Linux has no heuristic to notice and give up on ECN; that is why I do not enable it toward paths I do not control. If you would rather not open the whole box, per-destination ip route ... features ecn is a finer knife; that is how I ran my tests, and I rolled them back without a trace.

Inside a datacenter, aiming for low latency. Algorithms like DCTCP force ECN through tcp_ca_needs_ecn() without consulting the sysctl; anyone wanting to try L4S needs tcp_ecn = 3 on 6.18 or later, a queue that understands ECT(1) and a congestion control algorithm built for L4S; tcp_ecn = 3 on its own only enables AccECN feedback, cubic does not send ECT(1), and Prague is not in mainline yet. That is the subject of the next lab, not this post.

Monitoring, in every case. Linux has no nstat counter for ECN; on my server nstat -az | grep -i ecn came back empty. The windows you have are ss -tino, the ecn_mark counter in tc -s qdisc, and the ToS column of tcpdump -v. Read ss without -o and you will see zero, like I did, and suspect the wrong thing.

The measure of an open door

The value of a sysctl says that a feature is permitted, not that it is working. tcp_ecn = 2 has kept the door open for twenty-five years, and on my server one stranger knocked in three minutes, then stopped marking once inside. That does not mean ECN has failed; most of the large networks I tried say yes in the SYN-ACK, Apple asks, the Windows templates say "Enabled", and the kernels are ready for the second attempt. But in a mechanism where the word "on" is spread across four separate doors, what needs counting is not what is open but what gets through. In my count, 7 of 269 connections ever saw a marked packet, all seven from my own Mac; the one stranger who got through the negotiation stopped at the door.

The macOS source files mentioned in the body live in Apple's XNU repository: bsd/netinet/tcp_output.c (the sysctl handlers and tcp_set_ecn) and bsd/netinet/tcp_cache.c (the ECN back-off heuristic). My other connection-level measurement posts: TIME_WAIT and tcp_fin_timeout, who actually reads the keepalive setting.

DEV Community