Mustafa ERBAY

Posted on Sep 16 Originally published at mustafaerbay.com.tr

TCP-AO: Changing the Key While the Connection Is Open

#tcp #tcpao #bgp #linux

The password mechanism protecting BGP sessions dates from 1998: RFC 2385, an MD5 of the TCP segment. In the years the attacks on MD5 were being published, the IETF wrote the document to replace it, RFC 5925, the TCP Authentication Option; that was 2010. Getting into Linux took thirteen more years: Dmitry Safonov's patch set merged in 6.7, net/ipv4/tcp_ao.c does not exist in 6.6 and does in 6.7. My server's kernel is 6.8, CONFIG_TCP_AO=y; the 6.8 series has carried AO since March 2024, it has been sitting on this machine since it was installed in May, no application uses it, and its counters have never moved off zero. BIRD has spoken it since 2.17 (April 2025); in FRR the request opened in October 2020 is still open, the first patch from 2021 was closed unmerged, and TCP_AO_ADD_KEY does not appear in the bgpd source today. "Kernel ready, daemon not".

Rather than wait for a tool, I asked the kernel directly. Two network namespaces, a veth pair, a setsockopt structure filled in by hand from Python, and tcpdump. The aim was to answer three questions: what does a signed TCP connection look like on the wire, what does a wrong key do, and can the key change without the connection dropping.

What MD5 cannot do

The essence of the comparison table in the kernel documentation: TCP-MD5 carries one key per connection; changing it requires both sides to change at the same moment within the MSL, which in practice means dropping the session; no protection against ICMP "hard errors"; no protection against replayed segments. In TCP-AO the number of keys is many, each key has a SendID/RecvID pair, switching the active key is part of the protocol (RNextKeyID), ICMPs are ignored by default on established connections, and the Sequence Number Extension (SNE) gives the MAC a 64-bit sequence number, so a segment replayed from a previous wraparound cannot pass verification. The MAC is 12 bytes instead of 16: HMAC-SHA-1-96 and AES-128-CMAC-96 are mandatory, and Linux additionally recognises HMAC-SHA256. Something is lost too: connectionless RST. An AO RST needs the ISNs to be signed; someone without the session cannot produce an RST, which is exactly what is wanted.

Filling the structure by hand

Python's socket module has no TCP_AO_ADD_KEY constant; its number is 38, the structure is struct tcp_ao_add, 288 bytes: the peer's address inside a 128-byte sockaddr_storage, a 64-byte algorithm name (hmac(sha1), in kernel crypto API spelling), an ifindex for VRF, the set_current/set_rnext bits, prefix, sndid, rcvid, maclen, flags, key length and an 80-byte key. I built it with struct.pack and sent it; the kernel checks the structure byte by byte, so the reserved fields must be zero.

TCP_AO_ADD_KEY = 38
addr = (pack("H", AF_INET) + pack("!H", 0) + inet_aton(peer) + bytes(8)).ljust(128, bytes(1))
buf  = addr + b"hmac(sha1)".ljust(64, bytes(1)) \
     + pack("iIHBBBBBB", 0, flags, 0, 32, sndid, rcvid, 12, 0, len(key)) + key.ljust(80, bytes(1))
sock.setsockopt(IPPROTO_TCP, TCP_AO_ADD_KEY, buf)   # len(buf) == 288

The IDs are directional: in this first run sndid=200, rcvid=100 on the client, the reverse on the server (in the rotation run I will give the IDs the other way round, 100→200). The first key is added before connecting or listening; adding the first AO key to an established connection afterwards is forbidden, and so is removing the last one; further keys can be added to an established connection, which is what rotation relies on.

On the wire

With matching keys the connection came up instantly; nine segments, all nine signed. The SYN's option field:

options [tcp-ao keyid 200 rnextkeyid 100 mac 0xe194beeea1ab744d474dbdb2,mss 1460,sackOK,TS val 3418956430 ecr 0,nop,wscale 7]
1d10 c864 e194 beee a1ab 744d 474d bdb2

Kind 29, length 16, KeyID 200, RNextKeyID 100, a 12-byte MAC. tcpdump decodes the option by name (since 4.9). nstat in the server-side namespace: TcpExtTCPAOGood 6; there are per-socket counters as well, read with getsockopt(TCP_AO_INFO).

Putting the same SYN next to TCP-MD5 turned up a difference that is easy to miss:

options [nop,nop,md5 ...,mss 1460,nop,nop,sackOK,nop,wscale 7]

No timestamp. My first thought was a space calculation, but the arithmetic does not hold: MD5 20 (18 plus two nops), MSS 4, a 12-byte TS+sackOK block in which sackOK takes the TS's nop slots, and wscale 4, exactly 40; it would have fit. The kernel source gives the reason elsewhere: tcp_syn_options says timestamps = false as soon as it sees an MD5 key, unconditionally. The rationale is not the SYN but the data segment: on an established connection MD5 20 + TS 12 = 32 bytes, and a single SACK block (12) does not fit in the remaining 8; Linux prefers SACK over timestamps with MD5. With AO, 16 + 12 = 28, and one SACK block fits in the remaining 12; RFC 5925 §7.6 does the same arithmetic. In the keepalive post I described how RTT measurement and PAWS rely on the timestamp; on Linux, MD5-protected BGP sessions have gone without both for years, AO-protected ones do not.

The wrong key

I gave the client a different key:

server: accept TIMEOUT
client: FAILED after 6.0s: TimeoutError
TcpExtTCPAOBad 6

Six SYNs, zero SYN-ACKs, zero RSTs. The kernel drops the wrongly signed segment and tells the other side nothing; from the client's point of view this is indistinguishable from a firewall DROP. Six SYNs fitting into six seconds is a newer kernel detail too: tcp_syn_linear_timeouts=4 (since 6.5) makes the first four retransmissions one second apart instead of exponential backoff, and the fifth is 2⁰×RTO, one second again (RTO sequence 1, 1, 1, 1, 1, 2, 4…); the tcpdump timestamps are 1.06 / 1.02 / 1.02 / 1.02 / 1.03 seconds apart.

Every drop writes a rate-limited line to dmesg. Over the lab 42 lines accumulated; including the "required" case I ran twice (12) and a wrong MD5 password I tried for comparison (6), the distribution matches the runs:

12 AO hash is required, but not found
 6 AO hash mismatch
12 AO key not found
 6 MD5 Hash failed
 6 Unexpected MD5 Hash found

I tried other cases too; all were dropped. Key on the client only, server without a key: TCPAOKeyNotFound 6. The RFC's flexibility of "a connection matching no MKT may ignore AO" does not exist in Linux; if the socket has no AO info, tcp_inbound_ao_hash drops the signed segment as AOUNEXPECTED. The reverse: key on the server, plain SYN from the client: TCPAORequired 6. I did not need to switch on the ao_required flag for that; if an MKT exists for the peer, unsigned segments from that peer are not accepted, the flag is for peers without an MKT. The last was the mixed world: an AO client against an MD5 server gives KeyNotFound, an MD5 client against an AO server TCPMD5Unexpected. The documentation says so: both kinds of key may be added to a listening socket, but not both for the same peer; a BGP daemon can take both kinds of neighbour on the same port during a transition, but not the same neighbour with both kinds.

Key rotation

On to the thing MD5 cannot do. I added two keys on both sides: the first pair (client 100→200) active, the second (101→201) standby. During a five-message ping-pong, after the second message the client said current=101, rnext=201 via setsockopt(TCP_AO_INFO). On the wire:

[P.] keyid 100 rnextkeyid 200      client, old key
[P.] keyid 200 rnextkeyid 100      server, old key
[P.] keyid 101 rnextkeyid 201      client rotated, asks the server for 201
[P.] keyid 201 rnextkeyid 100      server moved to 201, but still asks the client for 100
[.]  keyid 100 rnextkeyid 201      client went BACK to 100

On the first attempt the rotation stayed half done, and the reason is instructive. The server saw the client's RNextKeyID request and switched its sending key to 201; but the server's own rnext had stayed at 100, so on every segment the server kept saying "write to me with 100". The client's kernel obeyed the peer's request and went back from 101 to 100. The socket counters say the same: on the client, current: 100, rnext: 201. This is what the RFC calls "backing up", and the sentence in the documentation is clear: RNext changes only by user intervention; the kernel does not automatically request what the other side requested.

On the second attempt the server also said current=201, rnext=101 after the second message:

[P.] keyid 101 rnextkeyid 201
[P.] keyid 201 rnextkeyid 101
[.]  keyid 101 rnextkeyid 201

From then on every segment went with the new pair, the bad counters stayed at zero, the connection did not drop. Before the fifth message both sides deleted the old key with TCP_AO_DEL_KEY (a 144-byte struct tcp_ao_del); the connection carried on. With MD5 this operation meant dropping the session and rebuilding it; with AO it is four setsockopts and zero lost packets. But the order matters: first the new key on both sides, then one side's rnext, then the other side's rnext, then deletion. A one-sided rnext ping-pongs; the documentation deliberately leaves key management to user space ("policy decisions"), and this ordering is the daemon's job.

The buffer getsockopt reads

While reading the counters I fell into a trap. getsockopt(TCP_AO_INFO) worked on the client and returned EINVAL on the socket the server had accepted. The kernel source explains why: tcp_ao_get_sock_info first reads the buffer the user passed, and returns EINVAL if the reserved fields are not zero; so that the structure can be extended later. Python's socket.getsockopt(level, opt, buflen) hands over the buffer without zeroing it; on the client it happened to be zero, on the server it was not. Passing a zeroed create_string_buffer(48) via ctypes made both work. The assumption "getsockopt only writes" is wrong here; no problem for code going through libc, but in higher-level languages you have to look at what is in the buffer.

The limits Linux draws

There are a few limits the documentation states plainly and that I saw indirectly in the lab. MKT matching is by peer address only (plus, optionally, prefix and VRF ifindex); the RFC's flexibility of matching by port ranges is not implemented; since keys belong to a socket, you cannot say "on this listener, only this source port of that neighbour", every connection from that address to that socket wants the same key. HMAC-SHA256 is a Linux extension: it uses the same KDF path as HMAC-SHA1, without the entropy extraction RFC 5926 defines for AES-CMAC, and expects the key to be supplied with full entropy; unless the other end follows the same path it will not interoperate, so for a session with another vendor's box, staying on one of the two mandatory algorithms is the safer choice. A key change on a listening socket is not reflected in connections waiting in the accept() queue; the documentation delegates this to user space and expects the daemon to check the keys of the socket returned by accept(). And the ignoring of ICMPs on established connections: unless accept_icmps is switched on, destination-unreachable messages do not bring the session down, the counter is TCPAODroppedIcmps; this closes the door on knocking BGP sessions over with ICMP, but noticing a neighbour that has really died is now up to keepalive.

From MD5 to AO

Translating the lab's ordering to a BGP neighbourship, the plan is four steps. First an AO-capable daemon and a 6.7+ kernel on both ends; in BIRD, authentication ao and a keys { key { secret …; algorithm cmac aes128; send id …; recv id …; } } block (the Send/Recv IDs are exactly the kernel's sndid/rcvid; rotation goes through preferred/deprecated marks, and the documentation notes that only the selected key is used during the initial handshake), in FRR not yet; ExaBGP 6.0 also speaks AO on Linux. Then the transition window: since MD5 and AO keys cannot both be added for the same neighbour on the listening socket, the neighbourship is dropped once and rebuilt with AO; this one unavoidable outage is the last of the price paid at every key change in the MD5 era. Third, rotation discipline: the new SendID/RecvID pair is added on both ends, one side requests rnext, the other switches current and moves its own rnext to the new key as well, then the old key is deleted on both ends; you saw a one-sided rnext ping-pong. Fourth, monitoring: if TCPAOBad and TCPAOKeyNotFound in nstat move off zero, either the keys mismatch or someone is writing to the wrong neighbour; the TCP: lines in dmesg give the source address and flags (the KeyID only in the AO key not found line), so which neighbour was dropped reads off one line.

Who it is for

The kernel documentation does not hide the audience: "mainly intended users are BGP processes, not any random applications". Key management is deliberately in user space, not the kernel; there is no ip tcpao add key command; the documentation compares the two approaches and says it chose setsockopt. So AO's pace in the field depends on the daemons: BIRD and ExaBGP have it, FRR does not, router vendors are a separate story. On my own server the only candidate for AO would be a session of the kind I described in the BGP post; there is no BGP here.

For anyone wanting to try it: kernel 6.7+, CONFIG_TCP_AO, the same algorithm and maclen on both ends, directional sndid/rcvid, the first key before connecting. On AO sockets the kernel disables GSO (sk_gso_disable), worth measuring on a high-volume session. TcpExtTCPAO* in nstat and TCP: lines in dmesg are enough for diagnosis; and do not let a rotation coincide with a handshake.

Measurements on Ubuntu 24.04, kernel 6.8.0-139-generic, tcpdump 4.99.4, Python 3.12, veth between two ip netns; structure layouts counted by hand from include/uapi/linux/tcp.h (tcp_ao_add 288, tcp_ao_del 144, tcp_ao_info_opt 48 bytes). For BIRD support, NEWS 2.17; for FRR's status, #7240 and the closed PR #9442, as of 16 September 2026. For the operator side, DENOG's TCP-AO guide.

DEV Community