Divy Yadav

Posted on Aug 22 Originally published at yadavdivy296.Medium

Where Sandbox Ingress Speed Actually Comes From

#webdev #programming #ai #architecture

A proxy chain into an isolated sandbox usually has more than one hop, and each one deserves its own answer to the same question:

Does this specific layer need to understand the application protocol, or is it mainly moving bytes between two points that already trust each other?

Many production proxy chains combine L7 and L4 functions at different hops. The interesting engineering work is in separating the cost of parsing and buffering at L7 from the cost of moving bytes at L4.

Get that separation wrong, and a technique like kernel Transport Layer Security (kTLS) can end up credited with a win that actually came from removing a parser.

A recent ingress rebuild provides a concrete test of that separation. The team moved one dataplane hop from a full L7 reverse proxy to an L4 forwarder using kTLS and splice(2).

They measured each change on its own rather than only comparing the old system to the new one. Most of the CPU saving in that staged comparison came from removing the parsing layer; kTLS and splice(2) added a smaller, separate throughput gain on top.

Tensorlake’s rebuild is the worked example; the underlying decision applies to proxy chains more broadly.

The questions underneath it, when an L7 hop earns its cost, when L4 becomes attractive, what disappears when a hop stops parsing, what has to be rebuilt at L4, and how to test whether removing L7 actually matters, apply to any proxy chain, not just this one.

The real question: which hop needs to understand the protocol

The path into a Tensorlake sandbox runs through an edge gateway and a separate dataplane-side hop. The edge gateway, built on Cloudflare's Pingora, terminates the client's Transport Layer Security (TLS) connection, accepts HTTP/S, WebSocket, and gRPC traffic, and extracts sandbox-routing information from the request for those protocols. SSH uses a different extraction path from the connection.

That hop is still L7 today, and stayed that way through the entire redesign. What changed sits behind it: a dataplane-side hop that used to run a full L7 reverse proxy speaking HTTP/2 over mutual TLS (mTLS), and now runs an L4 forwarder.

That's the shape of the actual decision, and it's made per hop, not once for the whole path: which layer needs to read the application protocol, and which layers only need to move authenticated bytes?

Keep L7 where a hop needs request-aware behavior, such as protocol-aware routing, application-aware retry policy, request deadlines, or synthesized responses. L4 becomes a candidate when an earlier hop has already made those decisions and the next hop only needs to move bytes over an authenticated channel.

The team also considered lower-level alternatives for that dataplane hop, including VXLAN, eBPF-steered routing, a CNI plugin, and host-level encapsulation. The engineering post says the team chose a non-invasive, application-layer mechanism instead, because the platform runs across AWS, GCP, and planned GPU neoclouds.

The data path shouldn't depend on any one provider's networking primitives. L4 forwarding won out over L7 for this one hop; the choice against those lower-level options was a separate portability decision.

The change comes down to one hop:

OLD - two L7 hops between the client and the app

+--------+      +--------------+      +----------------+      +-------------+
| Client | -->  | Edge gateway | -->  |    L7 proxy    | -->  | Sandbox app |
|        |      |  TLS + auth  |      |  mTLS+HTTP/2   |      |             |
+--------+      +--------------+      +----------------+      +-------------+


NEW - one L4 hop between the client and the app

+--------+      +--------------+      +----------------+      +-------------+
| Client | -->  | Edge gateway | -->  |  L4 forwarder  | -->  | Sandbox app |
|        |      |  TLS + auth  |      | kTLS+splice(2) |      |  plaintext  |
+--------+      +--------------+      +----------------+      +-------------+

Only the third box changes. The edge gateway stays L7 in both paths.

The public-facing side of that edge layer is documented separately, in the networking docs: the proxy preserves the request path and query string, supports WebSocket upgrades, and forwards gRPC over HTTP/2. Those are request- and protocol-aware behaviors, the kind of work that generally belongs at an L7 boundary.

When L7 is worth keeping

An L7 hop earns its cost by reading the request. Because it can see what's actually in flight, it can support:

routing on a path, header, or other application-level field
an application-aware retry policy
a per-request timeout or deadline
transforming a request or response in flight
a synthesized, application-level error when something downstream misbehaves

The edge gateway here is a clean example of a hop where that trade still holds. For HTTP/S, WebSocket, and gRPC, it extracts routing information from the request itself and authenticates the user; SSH uses a different extraction path from the connection.

An L4 hop cannot perform those application-protocol-aware functions without parsing the traffic, although it can authenticate its own transport channel using mechanisms such as mTLS.

Nothing about the dataplane redesign touched that layer, because the argument for dropping L7 never applied to it.

When L4 starts looking attractive

A hop becomes a real candidate for L4 once its cost is dominated by parsing and buffering bytes it never needs to act on, and that cost scales with data volume rather than request count. That's a different profile from a hop handling many small, distinct calls, where the request-level work is the entire reason the hop exists.

The dataplane hop here fit that profile closely. Traffic included thousands of small calls to start, stop, and observe sandboxes, alongside file uploads and downloads that moved substantially more data.

The engineering post says the old dataplane proxy parsed and buffered that bulk traffic even though the hop never needed to interpret it, since routing had already happened at the edge.

There was also a separate, structural problem layered on top. That proxy shared a binary with the sandbox orchestrator, so a routine orchestrator deploy could interrupt live connections that had nothing to do with the update.

That's a separate issue from the performance question: a team facing only the deploy-interruption problem could address it by decoupling the proxy's lifecycle from the orchestrator's, without changing the network layer at all.

What you give up when a hop drops to L4

Nothing about moving a hop to L4 is free. An L4 proxy that does not parse the application protocol can forward bytes but cannot make request-semantic decisions, so the removed L7 hop no longer provides:

a different routing decision for each request on the same connection
intelligent, application-aware retries
a real, synthesized error response when something downstream fails
watching individual requests to know a connection is still active

Any capability that depended on reading traffic has to move somewhere else or get rebuilt on a signal the L4 layer can actually observe. This forwarder takes on two of those: routing and liveness.

Routing information, which used to be obtained from each request, now arrives before tenant data in a short, length-bounded preamble naming the sandbox ID and the sandbox’s private target address (ip:port). The gateway learned that target from the scheduler.

That happens over a connection that's mutually authenticated in both directions: the forwarder presents a certificate that the gateway pins, and the gateway presents a client certificate whose identity the forwarder checks against an allowlist.

Liveness, which used to come from watching requests, is rebuilt in this implementation by metering bytes moving through each connection and reporting those byte counts to the layer that owns the sandbox’s idle timer.

In Tensorlake's implementation, that idle timer runs on a threshold rather than a fixed lifetime: the forwarder meters bytes and reports them to the dataplane, which resets the sandbox's idle timer as long as traffic keeps flowing, and only lets it time out once nothing has moved for the configured period.

Both routing and liveness took real engineering work. An L4 hop doesn't produce either one on its own.

Why kernel TLS enables the zero-copy path, but isn't where most of the performance gain comes from

The forwarder still terminates mutual TLS on the gateway connection, which is what makes the routing preamble trustworthy in the first place. A plain L4 forwarder that does this the ordinary way, decrypting into a userspace buffer and writing the plaintext back out, already gets most of the benefit of dropping the L7 hop.

This setup measured 2.07 GB/s and 0.50 CPU-seconds per GB at exactly that stage, before kTLS entered the picture at all. Removing the request-parsing layer is what did that.

The team's own expectation going in was that kTLS would be responsible for most of the savings, that eliminating two userspace copies was where the CPU went. The isolated measurement said otherwise.

What kTLS adds on top is narrower than it first sounds, and it's worth being precise about what kind of narrower. In a conventional userspace TLS termination path, encrypted bytes cross into userspace, are decrypted there, and the resulting plaintext is then written onward, creating the userspace staging and boundary-crossing work that this design aims to avoid.

Kernel TLS moves symmetric TLS record processing into the kernel after the userspace handshake and key installation, so application data is decrypted on receive and encrypted on transmit by the kernel. In this socket-to-pipe-to-socket implementation, that enables splice(2) to avoid the ordinary userspace staging path for tenant payloads; it should not be read as a guarantee that every possible internal copy is eliminated in every kTLS configuration.

The plain userspace-copy stage above still had to copy every decrypted byte through the application; it just did that without also parsing a request first. That alone is what produced most of the measured gain in this staged test, before kTLS entered the picture.

What kTLS specifically supplies is a way for a hop to keep terminating TLS, which the routing preamble depends on, while still avoiding the ordinary userspace staging path in this implementation, instead of requiring a userspace copy for every decrypted byte because the hop needs to inspect the TLS record layer.

Tensorlake reports requiring Linux with CONFIG_TLS and kTLS receive support and using Linux 6.x in production; it reports Linux 5.1 or newer for its TLS 1.3 path. Actual availability depends on the kernel build and configuration, distribution, TLS-library integration, cipher support, and the socket path.

This implementation uses software kTLS, with cryptography on the CPU rather than required NIC offload.

It's also fail-closed by design: the forwarder daemon holds no sandbox state of its own and refuses to start at all if the kernel can't attach the TLS ULP, rather than silently falling back to a slower, userspace-copying path.

On the CPU that's left, the encryption itself is a minority cost.

AES-256-GCM ran at roughly 7.9 GB/s per core. At 0.49 CPU-s/GB, that implies approximately 0.127 CPU-s/GB for the cryptographic work, or about 26% of the measured forwarder CPU. The remainder was attributed to TCP handling and syscalls in that measurement; this should not be treated as a portable AES performance constant.

How to know whether removing L7 actually matters for you

Comparing only the old system to the new one won't tell you what caused the gain.

The useful version of this test has three stages: the existing L7 path, an L4 path that still does a plain userspace copy, and the L4 path with kTLS and splice(2) added on top. The middle stage helps isolate the effect of removing the parser and buffering layer from the incremental effect of the kTLS-plus-splice(2) path, provided the workload and measurement conditions remain controlled.

The staged measurements:

STAGED MEASUREMENT: isolate each change before crediting it

+--------------------+      +--------------------+      +--------------------+
|    L7, two hops    |      | L4, userspace copy |      | L4, kTLS+splice(2) |
|     1.12 GB/s      | -->  |     2.07 GB/s      | -->  |     2.50 GB/s      |
|   0.90 CPU-s/GB    |      |   0.50 CPU-s/GB    |      |   0.49 CPU-s/GB    |
+--------------------+      +--------------------+      +--------------------+

removing L7: +0.95 GB/s, -0.40 CPU-s/GB
adding kTLS + splice(2): +0.43 GB/s, -0.01 CPU-s/GB

Measuring those as separate steps is what makes it possible to say that removing the L7 hop accounted for most of the throughput gain and nearly all of the CPU saving, with kTLS contributing the smaller remainder.

That conclusion describes this one test. It isn't a claim that every L7-to-L4 migration will split the same way.

The test also didn't stop at a single connection, and it was direct about the limits of what it measured. Under concurrency, eight tunnels on that same host aggregated to 8.67 GB/s, at which point the bottleneck was serialization in the splice loop rather than CPU.

The headline numbers come from one connection, one direction, over loopback, on a single host, stated plainly as a bound on what this specific test can tell you rather than a production capacity claim.

A benchmark that changes routing, authentication, and workload shape all at once, and only reports one before/after number, won't tell you which change actually caused the result.

When this doesn't matter at all

This throughput optimization is unlikely to materially improve a hop dominated by short, frequent calls.

Start, stop, and observe calls were never bandwidth-bound, and kTLS adds little to them, since the handshake stays in userspace and dominates a short connection.

No end-to-end latency numbers have been published for the new path, so this piece doesn't infer any.

Those calls can still see a different failure model, unrelated to throughput.

A forwarder that never parses a response cannot synthesize an application-level status code. In Tensorlake’s implementation, a half-close is forwarded as a half-close and an upstream reset reaches the client as a reset, rather than being converted into a proxy-manufactured response or timeout. TCP defines the underlying half-close and reset signals, but transparent propagation is a property of this forwarder’s implementation.

A bandwidth gain and a latency gain are two different claims. A workload bound by connection setup rather than data volume can see close to nothing from this change, even while bulk transfers improve substantially.

The takeaway

Use this sequence on your own ingress path, independent of what any one platform did:

Find the hop that's actually costing you something, and check whether the cost is per-byte, per-connection, or per-request. Bulk transfers expose parsing, buffering, and copy costs. Short calls stay dominated by connection setup and the handshake.
Name what that hop currently does that depends on reading traffic: routing, retries, deadlines, liveness, honest errors. All of it has to move elsewhere or get rebuilt on a signal the lower layer can actually see.
If the hop still needs to authenticate its traffic, work out separately whether you can keep that authentication without paying for a userspace copy on every byte. That's a distinct problem from removing the parsing layer, and conflating the two is how a technique like kTLS ends up credited with a win it didn't cause.
Benchmark the removal in three stages under concurrency, before you believe the number.

If the hop in question is mostly handling small, frequent calls, this is often a lower-priority throughput optimization unless measurements show a meaningful per-byte or proxy-CPU bottleneck. The redesign may still be worthwhile for lifecycle isolation or a cleaner failure model.

DEV Community