gRPC over QUIC: faster seeks, slower bulk

Abstract. Notes from testing whether to move KeibiDrop, a post-quantum encrypted peer-to-peer filesystem, from gRPC over TCP to gRPC over QUIC. The product reads remote files on demand, so the goal is to survive an IP change, keep seek latency low, stay correct, then go fast, in that order. Measured on a 12-core macOS laptop, a 4-core Linux VPS, and a real link between a laptop in Barcelona and the VPS at 72 ms round trip. Findings: on macOS the QUIC transport is syscall-bound at about 137 MB/s because the platform has no UDP segmentation offload, and the undocumented sendmsg_x batching syscall lifts it to about 700 MB/s; a teardown deadlock traced to closing a QUIC stream concurrently with a write, fixed with a stream reset; connection migration moves a live gRPC channel to a new socket, verified with packet counters; on the real WAN a single QUIC stream loses to TCP on bulk transfer by up to 16x under Wi-Fi loss, but a random seek under saturating prefetch lands in 109 ms on QUIC against about 3 seconds on TCP, a 28x difference, because the seek gets its own stream instead of queuing behind the prefetch. The design that follows: the interactive on-demand path over QUIC for seek isolation and migration, bulk on lossy links over TCP.

The question, and what we optimize for

KeibiDrop is a peer-to-peer filesystem. You mount a remote peer's files and read them on demand. Open a 40 GB video, seek to the middle, and only the bytes you touch cross the wire. The transport underneath is gRPC over our own encrypted TCP connection.

TCP has one problem we cannot fix from userspace. When the 5-tuple dies, the connection dies. A laptop that moves from café Wi-Fi onto a phone hotspot changes its IP, and the transfer stops. QUIC identifies a connection by a connection ID rather than the 5-tuple, so it can migrate across an address change with the stream intact. We wanted to know whether gRPC could run over QUIC and keep that property.

We optimize for the experience of an on-demand filesystem, not for bulk throughput in MB/s. The order of priorities is: survive an IP change, keep seek latency low, stay correct, and then speed. All the numbers below are measured. Where a number surprised us, we looked for the cause before believing it. The hardware is a 12-core macOS laptop, a 4-core Linux VPS, and a real link between a laptop in Barcelona and that VPS at 72 ms round trip.

Getting gRPC onto a QUIC stream

gRPC already speaks net.Conn. In KeibiDrop it runs over our encrypted SecureConn, which is a net.Conn wrapping a net.Conn, so QUIC slots in underneath the same seam. The adapter that makes one QUIC stream look like a net.Conn is small, because a quic.Stream is already a reliable, ordered byte stream with no framing to add:

type quicConn struct {
    *quic.Stream            // Read/Write/deadlines come from the stream
    conn *quic.Conn         // for the two address methods the stream lacks
}

func (c *quicConn) LocalAddr() net.Addr  { return c.conn.LocalAddr() }
func (c *quicConn) RemoteAddr() net.Addr { return c.conn.RemoteAddr() }

One detail is worth stating now, because it returns in the deadlock section. quic.Stream.Close() is a graceful, send-only half-close, and quic-go documents that it must not be called concurrently with Write. A net.Conn.Close has to tear the whole connection down. The correct call is CancelWrite, a stream reset, not Close. We used Close first and paid for it later.

With the adapter, a listener that hands one stream up as a conn, and gRPC's NewClient pointed at a passthrough:/// target so it skips DNS, a gRPC call completes over QUIC. Unary, server-streaming, and concurrent calls all work, verified byte for byte and clean under the race detector. gRPC's own TLS stays off. QUIC's mandatory TLS 1.3 sits underneath with a throwaway certificate, because the real confidentiality is a post-quantum handshake we add on top (ML-KEM-1024 and X25519, the same primitives KeibiDrop already uses; QUIC's TLS 1.3 is not post-quantum and ours is).

Throughput on macOS

The first benchmark on the macOS laptop streamed a large file both ways on loopback:

transport	throughput
gRPC over TCP	1650 MB/s
gRPC over QUIC	140 MB/s

Twelve times slower. That is large enough to want an explanation, so we profiled it. A raw QUIC stream with no gRPC gave 137 MB/s, so the gRPC layer was not the cause; the QUIC transport was. The CPU profile showed where the time went:

56.67%  syscall.rawsyscalln          # the syscall itself
33.33%  ...sendmsg   (one WriteMsgUDP per ~1400-byte packet)
23.44%  ...recvmsg
 0.6 %  crypto (AES-GCM)             # negligible

Fifty-seven percent of the time was in sendmsg and recvmsg. Crypto, which people assume is the cost of an encrypted transport, was 0.6 percent. The reason is that macOS has no UDP Generic Segmentation Offload. On Linux, quic-go hands the kernel a 64 KB buffer and the kernel splits it into packets, so one syscall sends dozens. On macOS there is no such offload, so quic-go sends one sendmsg per 1400-byte packet. At 137 MB/s that is about 100,000 send syscalls per second and as many receives.

We checked this by running the same code on the Linux VPS, which has GSO. The VPS is the slower machine (its loopback TCP is 3.4x slower than the laptop's), and its QUIC still ran at 272 MB/s against the laptop's 137. Normalized against each machine's own TCP, QUIC went from 2 percent on macOS to 14 percent on Linux. So the macOS number is a platform limitation, not a design or crypto cost.

Batching sends with sendmsg_x

macOS has no GSO and no sendmmsg. The syscall-batching primitives that fix this on Linux do not exist here. It does have sendmsg_x, a Darwin syscall that sends many datagrams in one call, similar to Linux sendmmsg. It is undocumented. The syscall number is in the SDK (SYS_sendmsg_x = 481), but the struct msghdr_x it takes is not in any header. You declare it yourself from the XNU source and get the byte layout right, or the kernel reads garbage.

We wrote a small program that prints the struct offsets, checks them with a real batched send the receiver verifies, and benchmarks batch size against raw UDP send throughput:

batch	syscalls	throughput
1 (one per packet)	500,000	390 MB/s
8	62,500	675 MB/s
32	15,625	701 MB/s

Batching lifts the ceiling from 390 to about 700 MB/s and cuts the syscall count by 32 times. It levels off at batch 32, where the cost moves from syscall count to per-packet kernel work. So there is a real macOS lever here, about 1.8 times, with no root access. It helps in a second way. The 137 MB/s QUIC transfer used about five CPU cores, all of it the per-packet work TCP hands to the kernel, and in KeibiDrop that CPU competes with the FUSE filesystem, which is busy with encryption and chunking. Fewer syscalls per byte means fewer cores taken from the filesystem.

The primitive is now in our quic-go fork as the Darwin counterpart of the Linux GSO path, with the verified struct layout, ready to wire into the send loop.

The teardown deadlock

The test suite passed under the race detector and hung about half the time without it. That combination points at a timing-dependent bug rather than a random flake, because the race detector slows the code down. The goroutine dump named the test, a client cancelling a server-stream, and showed Server.Stop blocked while a QUIC write stayed stuck.

Two guesses were wrong, and both were quick to disprove by measuring. Sizing the download to fit the flow-control window did not help; it still wedged. A short idle timeout had no effect, because a connection that is actively trying to send is not idle. Reading the source settled it.

gRPC's http2Server.Close closes the underlying net.Conn, which is our quicConn.Close, which called quic.Stream.Close(), the graceful half-close from the adapter section, the one quic-go says must not be called concurrently with Write. gRPC's writer goroutine was concurrently blocked in a Write on that stream. The contract violation meant the write never returned, the conn never finished closing, and Server.Stop waited forever.

func (c *quicConn) Close() error {
    c.Stream.CancelRead(0)   // STOP_SENDING
    c.Stream.CancelWrite(0)  // RESET_STREAM: safe against a concurrent Write, and unblocks it
    return c.conn.CloseWithError(0, "")
}

Replacing Close with CancelWrite turned a 50 percent hang into ten green runs out of ten. The general point: when the race detector flips a test from pass to fail, look for the timing bug rather than adding sleeps. And a lightweight atomic counter is better than a print for diagnosing one, because the print changes the timing you are measuring. Ours made the test pass for a while and pointed us the wrong way.

Migration

Migration is the reason we started. quic-go exposes it directly: AddPath a new socket, Probe to validate the path, Switch. Getting it to fire took reading quic-go's own test. The client has to keep sending after the switch so the peer migrates too, and you must not close the old transport to prove it, because that tears the connection down.

We checked it with per-socket packet counters, moving a live gRPC channel to a new UDP socket mid-transfer:

migration: local addr 127.0.0.1:52598 -> 127.0.0.1:59126
path 1 writes +0        # old path goes silent
path 2 writes +90, reads +92   # new path carries both directions

After the switch the old path sent nothing and the new path carried both directions. The gRPC channel kept running. On the real product this is a laptop changing networks during playback without a stall.

The real WAN test

Loopback measures a transport's overhead, not its behaviour on a real network. So we ran it for real: a laptop on hotel Wi-Fi in Barcelona, about 45 Mbps down and 72 ms round trip, downloading from the VPS. No emulation and no offload tricks, so both TCP and QUIC send ordinary packets end to end.

concurrent streams	QUIC	TCP
1	32.8 s	2.0 s
4	5.5 s	1.7 s
16	4.7 s	1.7 s

A single QUIC stream dropped to about 2 Mbps of a 45 Mbps link while TCP filled it. The cause is real Wi-Fi packet loss and quic-go's congestion control, which backs off hard, while Linux kernel TCP handles the same loss well. We had gone in half-expecting the common claim that QUIC does better on lossy links. On a real lossy link the opposite held. For plain bulk transfer on a bad link, TCP won.

On its own this result argues against QUIC. We nearly stopped here.

Seek latency under prefetch

An on-demand filesystem is not used by streaming a file end to end. It is used by seeking. You open a video, jump to a position, jump again. Each jump is a small read at a random offset while a prefetch pulls bulk data in the background. The number that matters is how long that small read takes while the prefetch saturates the link.

On TCP, the prefetch and the seek share one byte stream, so the seek's bytes sit behind the prefetch already queued in the send buffer. That is head-of-line blocking, and it is the playback stutter we had seen before as self-congestion. On QUIC the seek gets its own stream, and the library interleaves its packets with the bulk, so it does not queue behind it.

We measured it on the same Barcelona link. Thirty small 16 KB reads under a saturating bulk stream:

	p50	p99	max
QUIC (seek on its own stream)	109 ms	322 ms	387 ms
TCP (seek behind prefetch)	3066 ms	4827 ms	4924 ms

A seek lands in about 109 ms on QUIC, roughly one round trip. On TCP it takes about three seconds, behind the prefetch. For someone scrubbing through a video the first is usable and the second is not, and the gap is larger than the bulk-throughput gap from the previous section.

This fits the earlier results together. QUIC's advantage for us is latency isolation, a seek that does not wait behind a prefetch, together with migration, a transfer that survives a network change. For an on-demand seekable filesystem that trade is worth it. TCP can match the fast lane only with a second connection per read, which is the data-channels idea we had been circling; QUIC gives it per stream, with shared congestion control.

What we build

The measurements point at a specific design. The interactive on-demand path runs over QUIC. Seeks and prefetch on separate streams is where the seek result comes from, and migration comes with it. Plain bulk on a lossy link stays on TCP, where quic-go's congestion control loses. On fast, clean links the macOS syscall ceiling is real, and sendmsg_x batching is the lever that lifts QUIC to about 700 MB/s while freeing CPU for the filesystem. Control traffic such as heartbeats and metadata rides QUIC, since it is small and it is the channel that has to survive a network change.

None of this was obvious at the start, and several steps were the opposite of what we expected. Measuring, rather than reasoning from what QUIC is supposed to be good at, changed the answer twice: once when the slow macOS transport turned out to be a syscall cost with a fix, and once when the throughput test's verdict flipped as soon as we measured seek latency.

The experiments are small and self-contained: the gRPC-over-QUIC adapter, the sendmsg_x program, and the two WAN benchmarks.

---

Measured on a 12-core macOS laptop, a 4-core Linux VPS, and a real link between a laptop in Barcelona and the VPS at 72 ms round trip. Originally published at keibisoft.com/blog/grpc-over-quic-seek-latency.html.