DEV Community

Cover image for GRO Hid My TLS Handshake's Real Packet Split, Even Over a Real Network
Alex Georgiev
Alex Georgiev

Posted on AI-assisted

GRO Hid My TLS Handshake's Real Packet Split, Even Over a Real Network

My last post measured Caddy's new default post-quantum TLS handshake and found the ClientHello grows from 318 to about 1490 bytes, comfortably past what a single TCP segment can carry on a normal path. Two readers pushed back with real technical substance: one worked through the exact byte math for why that should force a two-segment split, another asked whether I'd actually captured that split happening on a real lossy connection, or just inferred it from message sizes. Fair question. I hadn't.

So I rented a small cloud VM and pointed a real client at it over the actual internet, a genuine external path with a genuine MTU-constrained hop in the middle, the part Docker couldn't give me. I used DigitalOcean for this one, mostly because it was the account I already had open, the smallest droplet runs about $0.006/hour and any other provider's cheapest VM would have done the same job here, this isn't about that specific host. The first capture off that real external server still showed the ClientHello as one packet. Not because the server did anything wrong, because of a Linux default I hadn't accounted for yet.

Why a real external host and not another container

The earlier test ran Caddy and the client as two containers on one Docker bridge, on one machine. That setup has a known failure mode for exactly this kind of measurement: Generic Segmentation Offload (GSO) lets the sending side hand an oversized buffer to the virtual interface and defer real segmentation to a point tcpdump never sees, because the two ends never cross an actual MTU-constrained link. A capture there can show a single packet even when a genuinely constrained path would have split it. The fix isn't a bigger test, it's a real physical hop: my laptop, over home internet, to a server that doesn't share a kernel with the client.

First real-path capture: GRO got there first

doctl compute droplet create pqc-realpath-test \
  --region fra1 --size s-1vcpu-512mb-10gb \
  --image ubuntu-24-04-x64 --ssh-keys <key-id>
Enter fullscreen mode Exit fullscreen mode

Caddy running in Docker with --network host so it binds the droplet's real interface, tcpdump -i eth0 capturing before I connect, then from my own machine:

echo | openssl s_client -connect <droplet-ip>:443 -servername caddytest.local -groups X25519MLKEM768
Enter fullscreen mode Exit fullscreen mode
19:45:11.887772 IP X.X.X.X.50482 > droplet.443: Flags [P.], seq 1:1491, length 1490
Enter fullscreen mode Exit fullscreen mode

One packet, 1490 bytes, on a link whose negotiated MSS was 1460. That should be impossible on a standard Ethernet path, and it was the same illusion the Docker test produced, just for a different reason.

What was actually happening: GRO, not Docker

ethtool -k eth0 | grep -E 'generic-receive-offload|tcp-segmentation-offload|generic-segmentation-offload'
Enter fullscreen mode Exit fullscreen mode
tcp-segmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
Enter fullscreen mode Exit fullscreen mode

Generic Receive Offload reassembles a run of contiguous incoming TCP segments into one buffer before handing it up the kernel network stack, and standard packet capture sits above that point by default. My laptop's real NIC genuinely sent two Ethernet frames onto the wire, some router genuinely carried two packets to Frankfurt, and the droplet's own kernel quietly stitched them back together before tcpdump ever got a look. A real cross-internet path to a real external server was not enough to see the real segment count. I had to turn the reassembly off.

ethtool -K eth0 gro off gso off tso off
Enter fullscreen mode Exit fullscreen mode

Second capture: two real segments

Same client, same command, offloads disabled:

19:46:38.145193 IP X.X.X.X.50529 > droplet.443: Flags [.],  seq 1:1449,    length 1448
19:46:38.145193 IP X.X.X.X.50529 > droplet.443: Flags [P.], seq 1449:1491, length 42
Enter fullscreen mode Exit fullscreen mode

1448 plus 42 is 1490, the same total, now genuinely arriving as two packets, close enough together that tcpdump's microsecond timestamps show them as simultaneous, but two distinct segments with two distinct sequence ranges. The classical X25519 ClientHello, captured the same way, stayed a single 312-byte packet the whole time, on or off:

19:46:37.060644 IP X.X.X.X.50528 > droplet.443: Flags [P.], seq 1:313, length 312
Enter fullscreen mode Exit fullscreen mode

That's the split confirmed for real: not inferred from a length field, not an artifact of two containers sharing a kernel, an actual two-packet arrival over the actual internet, and it only shows up once you know to look past your own NIC's default reassembly.

What this does and doesn't answer

This confirms the precondition under discussion: the post-quantum ClientHello genuinely crosses a real path as two TCP segments where the classical one doesn't, which is the fact the extra-round-trip argument depends on. It doesn't repeat the packet-loss experiment itself, I didn't inject loss on this live droplet connection, that would mean deliberately breaking my own SSH session to the same box mid-test for no good reason. The loss-driven stall (one lost second segment, no fast retransmit, fallback to a timeout) is the same mechanism I measured separately with tc netem on a local link, now paired with proof that the two-segment precondition is real, not a local-network artifact.

Run it yourself

Any provider's cheapest VM works for this, the only requirement is a real external host with its own kernel. The commands below are DigitalOcean's doctl because that's what I used, swap in whatever CLI or console your own provider gives you.

doctl compute droplet create pqc-test --region <region> --size s-1vcpu-512mb-10gb --image ubuntu-24-04-x64 --ssh-keys <key-id> --wait

ssh root@<droplet-ip> '
  apt-get update -qq && apt-get install -y -qq tcpdump docker.io
  printf ":443 {\n  tls internal\n  respond hello\n}\n" > /root/Caddyfile
  docker run -d --name caddytest --network host -v /root/Caddyfile:/etc/caddy/Caddyfile caddy:latest
  ethtool -K eth0 gro off gso off tso off
  tcpdump -i eth0 -w /root/cap.pcap -s 0 "tcp port 443" &
'

echo | openssl s_client -connect <droplet-ip>:443 -servername caddytest.local -groups X25519MLKEM768
echo | openssl s_client -connect <droplet-ip>:443 -servername caddytest.local -groups X25519

ssh root@<droplet-ip> 'pkill tcpdump; tcpdump -r /root/cap.pcap -nn'
Enter fullscreen mode Exit fullscreen mode

Ten minutes end to end, a droplet destroyed the moment the capture was pulled, and the total bill for the whole test came in under a cent. If you're checking a segmentation claim on your own infrastructure and the numbers look too clean, check ethtool -k before you trust the capture.

Top comments (0)