An engineering lesson from integrating a userspace TCP stack into Velora.
Kernel bypass has an appealing promise: reduce the overhead of the kernel networking path, move packet processing into userspace, and gain more control over how packets are received, processed, and transmitted.
That promise was one of the reasons we decided to integrate mTCP and AF_XDP into Velora, our C-based, event-driven TCP protocol engine.
We expected the new backend to give us a performance advantage. Instead, our first controlled comparison showed the opposite.
And that turned out to be an interesting result in itself.
The expectation
Velora's native backend uses Linux's TCP stack, non-blocking sockets, epoll, per-connection state, and custom ring buffers. Its protocol parser uses an incremental state machine with optimized paths for contiguous headers and payloads.
The alternative backend integrates mTCP, a userspace TCP stack designed around efficient packet I/O, connection management, and batching. AF_XDP provides the mechanism for redirecting packets from an XDP program into userspace through shared-memory rings.
The reasoning seemed straightforward: if the application controls more of the networking path and avoids some conventional socket-processing overhead, why shouldn't it be faster?
The important caveat is that kernel bypass is a mechanism, not a performance guarantee. The gain depends on the packet I/O mode, batching, hardware support, workload, and the costs introduced by the userspace implementation itself.
What we measured
We started with a controlled PING/PONG benchmark comparing the native backend with the mTCP + AF_XDP backend.
Both ran on the same machine, using the same client workload: 50 persistent connections, a pipeline depth of eight requests per connection, a two-second warm-up, and a five-second measurement window. We performed three alternating runs per backend.
The test topology used a Linux virtual Ethernet (veth) pair. It was useful for isolating the software paths, but it was not a physical-NIC benchmark.
The results were clear.
| Metric | Native backend | mTCP + AF_XDP |
|---|---|---|
| Median throughput | 219,015 PONG/s | 55,238 PONG/s |
| Throughput range | 217,050–226,708 PONG/s | 49,610–72,205 PONG/s |
| Median p50 RTT | 1.815 ms | 0.606 ms |
| Median p95 RTT | 2.588 ms | 11.060 ms |
| Median p99 RTT | 3.423 ms | 12.164 ms |
| Server CPU utilization, normalized to one core | 95.6% | 47.7% |
Results from our three-run-per-backend benchmark; throughput and latency are application-level measurements.
Native Velora achieved approximately 3.97 times the median throughput of the mTCP + AF_XDP backend in this configuration.
Yet the results were not uniformly worse for AF_XDP. Its median round-trip latency was lower, while its high-percentile latency was substantially worse. It also used less of one logical CPU's capacity during the measured workload.
That combination matters. The userspace backend was doing less completed work per second despite not reaching the same measured CPU utilization as the native backend.
The measurements tell us what happened, but not yet precisely why.
Why the result makes sense
The first important detail was the interface.
Our benchmark used veth, not a physical NIC with native AF_XDP zero-copy support. The AF_XDP path ran in generic/SKB mode with copying into userspace. The Linux kernel's AF_XDP documentation explicitly distinguishes this mode from zero-copy operation.
Consequently, this experiment did not measure the full potential of an optimized, hardware-assisted AF_XDP deployment. It measured our current userspace backend under the interface and configuration we had available.
That distinction is fundamental.
A userspace stack must still receive packets, manage queues and buffers, process TCP state, handle retransmissions and timers, and transmit packets. Moving work out of the kernel does not eliminate that work. Depending on the path, it can replace existing costs with new ones.
The native backend, meanwhile, benefits from a mature kernel TCP implementation and the surrounding networking machinery. On this particular virtual interface and workload, that path was more effective.
The second detail was the workload itself. We tested complete request/response transactions rather than isolated packet reception. Each completed operation required a valid PONG to reach the client. That makes the measurement sensitive to the entire path—not just packet ingestion or protocol parsing.
We also observed a large difference between median and tail latency. AF_XDP's median RTT was lower, but its p95 and p99 were much higher. This suggests uneven completion times under the tested load. Queue handling, scheduling, buffering, or TCP processing are possible contributors, but further profiling is needed before assigning a specific cause.
The CPU figures reinforce the need for that investigation. Lower utilization is not automatically better efficiency if throughput has fallen even more. Equally, higher utilization does not imply an inferior design: the native backend was completing more than three times as many transactions per second.
The parser benchmark was a different experiment
There is another result worth separating from this comparison.
Velora's parser-oriented benchmark processes sequences of 64 PUBLISH messages followed by a PING marker. A sequence counts only after its confirming PONG reaches the client. With 10,000 loopback connections and 64-byte payloads, we reproduced approximately 2.44 million confirmed protocol messages per second in the native backend.
That result is not directly comparable to the 219,015 PONG/s result above. The parser-oriented test amortizes one response across many incoming messages; the PING/PONG test generates a response for every request. They measure different workloads.
This distinction is easy to lose when looking at performance numbers. A benchmark is only meaningful when we understand exactly what it counts.
What we learned
Our result does not demonstrate that mTCP is inherently slower than Linux TCP, or that AF_XDP cannot improve performance. The original mTCP research demonstrated substantial gains through coordinated optimization of the userspace TCP stack, packet I/O, and batching on its evaluated systems.
Our result is narrower: integrating mTCP and AF_XDP into an existing application did not, by itself, produce a throughput gain in our tested configuration.
That is still a valuable finding.
It tells us not to assume that an architecture associated with high performance automatically transfers those gains to a different application, machine, interface, and workload. It also gives us a concrete baseline for future work.
The next experiments would need to isolate the sources of overhead, verify the packet-processing path, examine queue and buffer behavior, and evaluate a NIC and driver that support native AF_XDP operation. We also need equivalent workloads and repeatable measurements before drawing broader conclusions.
For now, the native backend remains faster in the configuration we measured.
And that is a perfectly useful engineering outcome. We set out expecting an improvement, measured the result, and found that our assumptions did not hold for this implementation. The next step is not to defend the optimization—it is to understand the system.
Code and benchmark artifacts: Velora on GitHub.
Top comments (0)