DEV Community

BookMyHost
BookMyHost

Posted on Originally published at bookmyhost.com

Works on desktop, fails on phones: the post-quantum TLS bug hiding in your network card

#networking #security #devops #windows

Modern browsers now send a larger TLS hello. On some Windows servers, a receive-offload bug in the network card silently breaks about half of those connections. Here is how to spot it, why it happens, and the two ways to fix it.

The complaint

"The site opens fine on my laptop, but my phone says This site can't be reached." The server looks healthy: no errors in the web server logs, CPU normal, the firewall shows nothing blocked. Reload a few times and the page sometimes loads.

Symptoms

  • Browsers show ERR_CONNECTION_RESET, ERR_CONNECTION_TIMED_OUT, ERR_CONNECTION_ABORTED or "Check your internet connection", intermittently.
  • Phones notice it first; desktops seem fine.
  • The web server logs nothing for the failed attempts, because the request never gets that far.
  • curl or a TLS 1.2 test from your own network always works.
  • CDN/DDoS dashboards show no drops.
  • The server runs Windows Server with Receive Segment Coalescing (RSC) enabled on its network card.

What is actually going on

  1. The hello got bigger. Chrome, Edge and Firefox now add a post-quantum key exchange (X25519MLKEM768) to every TLS 1.3 handshake. That adds over a kilobyte to the ClientHello. At about 1.8 KB it no longer fits in one TCP packet, so it arrives as two segments.
  2. The card merges them, and gets one field wrong. RSC lets the network card combine back-to-back segments before Windows sees them. In the environment we observed (NVIDIA/Mellanox ConnectX-4 Lx, WinOF-2 driver 2.42), the merged segment arrived with the top bit of its TCP timestamp cleared.
  3. Windows then throws the rest away. TCP's PAWS check discards segments whose timestamp looks older than the last one accepted. With the top bit flipped, the second half of the ClientHello looks "from the past" and is dropped, on every retransmit. The handshake never completes.
  4. Why only half the time. Clients start TCP timestamps at a random value, so the top bit is set on about 50% of connections. Those are the ones that fail.
  5. Why "only phones". It isn't really a phone bug. Desktops tend to reuse an existing open connection, while phones open fresh ones constantly as radios sleep and networks switch. Older clients send a hello small enough to fit in one packet.

How to confirm it in two minutes

From any Linux machine outside your network (OpenSSL 3.5 or newer), compare a large and a small key share:

for i in $(seq 10); do echo | timeout 5 openssl s_client -connect YOUR_IP:443 -tls1_3 -groups X25519MLKEM768 2>/dev/null | grep -c "BEGIN CERT"; done
for i in $(seq 10); do echo | timeout 5 openssl s_client -connect YOUR_IP:443 -tls1_3 -groups x25519 2>/dev/null | grep -c "BEGIN CERT"; done
Enter fullscreen mode Exit fullscreen mode

If the first loop fails several times and the second never fails, this is your bug. A packet capture on the server shows the second segment dropped, and the first segment's timestamp exactly 2^31 away from the client's.

Fix 1: on the server

Disable RSC in Windows. It takes effect immediately, without resetting the adapter:

netsh int tcp set global rsc=disabled
Enter fullscreen mode Exit fullscreen mode

Also check your NIC vendor for a newer driver or firmware. In our tests, phone-style handshakes went from failing about half the time to 20 out of 20 succeeding.

Fix 2: at the network edge (what we did)

One server is easy. Finding every affected server in a datacentre is not. So our edge routers now remove the TCP timestamp option from incoming SYN packets. A connection that starts without timestamps never uses them, so there is nothing for the card to corrupt and nothing for PAWS to drop. On a Linux/nftables router it is one rule in a forward chain:

tcp flags syn / syn,ack,rst tcp option timestamp exists reset tcp option timestamp
Enter fullscreen mode Exit fullscreen mode

Two details matter:

  • Strip SYNs only, never SYN-ACKs. If traffic is asymmetric and only the reply were stripped, the two ends would disagree about timestamps, which is worse than the original bug.
  • The cost is small. You lose the timestamp option's finer RTT measurement and its protection for very fast single flows. On a busy edge router the CPU cost was too small to measure.

The takeaway

When "some users can't connect" and every log is clean, test the way your users connect: modern browser, fresh connection, post-quantum key share. More of the internet now sends handshakes that need two packets, and old offload bugs are coming back into view.

Originally published on the BookMyHost blog.

Top comments (0)