How the sidecar steals the traffic

A sidecar proxy only works if the application never knows it's there. This post is part 1 of a 2-part series about the three times the kernel reminded me how hard that is.

Section 1: The Problem

You build an application that runs in Kubernetes, and it runs fine for some time. Then you need to add telemetry, and your manager wants mTLS, circuit breaking, and retries. Easy, right? Except you have 60 applications running in your cluster, they all need these features, and there are other features sitting in the backlog. So you decide to build a proxy that sits in the pod and does all of this for them. But that means every application has to be reconfigured to send its traffic through the proxy. The fix is to make the proxy transparent: catch the traffic without the app knowing. In this post we will discuss how to make that happen.

Section 2: What a packet goes through

To start off, we need to know what iptables are. At its core, iptables is a command-line tool that allows you to configure the Linux kernel's built-in network filtering system.
The rules used to set up the Ananse iptables are as follows:

# inbound: anything arriving at the pod, except the sidecar's own ports
iptables -t nat -A ANANSE_PROXY_INBOUND  -p tcp --dport 15006 -j RETURN
iptables -t nat -A ANANSE_PROXY_INBOUND  -p tcp --dport 15021 -j RETURN
iptables -t nat -A ANANSE_PROXY_INBOUND  -p tcp -j REDIRECT --to-ports 15006
iptables -t nat -A PREROUTING -p tcp -j ANANSE_PROXY_INBOUND

# outbound: anything the app sends, with three escapes
iptables -t nat -A ANANSE_PROXY_OUTBOUND -m owner --uid-owner 1337 -j RETURN   # the sidecar itself
iptables -t nat -A ANANSE_PROXY_OUTBOUND -d 127.0.0.1/32 -j RETURN            # app talking to itself
iptables -t nat -A ANANSE_PROXY_OUTBOUND -d $KUBE_API_IP -j RETURN            # the API server
iptables -t nat -A ANANSE_PROXY_OUTBOUND -p tcp -j REDIRECT --to-ports 15001
iptables -t nat -A OUTPUT -p tcp -j ANANSE_PROXY_OUTBOUND

We have two primary chains: PREROUTING for inbound requests and OUTPUT for outbound requests. We have one set of rules for each. REDIRECT rewrites the destination to this same machine, both the address and the port, and RETURN says this request is allowed to skip the redirect and go directly to its destination (on the outbound side, this is the sidecar's own user, localhost, or the API server).
The first two inbound rules leave the sidecar's own ports alone: 15006 for traffic and 15021 for health checks and metrics. Everything else arriving at the pod gets sent to 15006.
On the outbound side, after the REDIRECT, the sidecar accepts a connection whose destination now reads 127.0.0.1:15001. (Inbound connections land on 15006, addressed to the pod's own IP.) The address the app actually wanted is gone from the packet. It survives in only one place; the kernel's conntrack table, and that is where the sidecar gets it back using SO_ORIGINAL_DST.

Section 3: The Gotchas

Like everything in engineering, there are gotchas we need to look out for. Here is the first one.

The Loop (UID 1337)

I hit this one early. I don't remember exactly what the first symptom looked like, but the cause was simple: intercepting all traffic leaving the pod meant the sidecar's own requests were redirected straight back into it. I had built an infinite loop.
The fix relies on how Linux handles processes. In Linux, every process runs under a specific User ID (UID). By convention in the service mesh world (borrowed from Istio and Envoy), the sidecar proxy is configured to run as a dedicated, unique user ID: 1337. Once I figured out the loop, the solution was to update our iptables script to check for this ID. If the traffic is owned by UID 1337, it is allowed out directly, bypassing the redirect back to port 15001.
To orchestrate this in Kubernetes, the injector sets the sidecar container to run as user 1337. The init container, which applies the iptables rules before the app starts, is different. Ananse's init container runs as root with NET_ADMIN and NET_RAW capabilities, because changing the kernel's netfilter rules is a privileged operation:

SecurityContext: &corev1.SecurityContext{
    RunAsUser:    ptr(int64(0)), // Root (required for iptables)
    RunAsNonRoot: ptr(false),    // Override pod-level runAsNonRoot
    Capabilities: &corev1.Capabilities{
        Add:  []corev1.Capability{"NET_ADMIN", "NET_RAW"},
        Drop: []corev1.Capability{"ALL"}, // Drop everything else
    },
},

Why Outbound Uses UIDs and Inbound Uses Ports

This loop fix highlights a crucial detail about how the Linux networking stack works. You might notice that our outbound escape rules check the user, while our inbound escapes check the port.
The kernel only knows who owns a packet when that packet is created on this specific machine. Packets arriving from the outside world have no local owner attached to them. Because of this, outbound rules can cleanly filter by the local UID, but inbound rules are forced to rely on the destination port to figure out where the traffic belongs.

Summary

Right now we have a working intercept. By using a combination of PREROUTING, OUTPUT, and a dedicated 1337 user ID, Ananse can transparently hijack both inbound and outbound pod traffic without a single change to the application's configuration.
The network layer is fully rigged, but the packet is currently sitting at our sidecar, stripped of its original destination. In Part 2, we will dive into the Go implementation to extract that address from the conntrack table, and look at two more things that broke: the deadlock that occurs when you run the control plane inside the very same mesh it manages, and the health probe that dialled itself.