When the mesh fights Kubernetes: a webhook deadlock, a probe loop, and what it costs

In part 1, we rigged the network so every packet in the pod passes through Ananse's sidecar, without the app knowing. That was the plan on paper. This post is about what happened when the plan met a real cluster.

Two bugs taught me the most. In the first, the mesh locked its own control plane out of the cluster. In the second, a routine health check made the sidecar call itself until it drowned in timeouts. After that, I'll cover what running the mesh actually costs, and what I'd change next.

The Deadlock (The Webhook Injecting Into Its Own Namespace)

In Ananse Service Mesh, the control plane runs a Mutating Admission Webhook. Crucially, this webhook is configured with failurePolicy: Fail. This is a deliberate security choice: if the webhook cannot be reached, pod creation is rejected. If we used Ignore, pods would silently start without a sidecar, breaking mTLS and routing without anyone noticing. We fail closed.
The normal injection flow works like this:

1. When Kubernetes creates a pod, the API Server intercepts the request.
2. The API Server looks at the MutatingWebhookConfiguration and says, "Before I create this pod, I need to send it to the Ananse Webhook to inject the init container and sidecar."
3. It sends an HTTP request to the webhook, gets the modified pod spec back, and creates the pod.

If you read those steps carefully, you might spot an issue waiting to happen.
The control plane itself runs as a pod. I learned exactly how bad this could be during a routine node drain when my cluster suddenly stopped accepting new workloads. If the control plane goes down for any reason, a fresh install, an upgrade, a node drain, or a crash, it has to go through this exact Mutating Admission flow to boot back up. The API server pauses the pod creation and sends the request to the webhook... but the webhook is the control plane pod that hasn't started yet. The API server gets no response, and after 5 seconds, it outright rejects the pod.
Because the pod is never created, nothing restarts. You are locked in a complete deadlock. And the stakes are high: while the control plane is down, every new pod in an injected namespace is refused.
We need a way to tell the API server, "If this specific resource comes to you, just let it through." In Kubernetes, you handle this using labels and selectors, specifically, a namespaceSelector in your webhook configuration.

Layer 1: The Immutable System Label (NotIn)

Instead of relying on admins to manually label namespaces to opt-out, we target the kubernetes.io/metadata.name label that Kubernetes automatically attaches to every namespace. Our namespaceSelector explicitly excludes critical infrastructure: NotIn [kube-system, ananse-system, kube-public, cert-manager]. Because Kubernetes manages this label, it is impossible for an admin to forget it.

Layer 2: The Explicit Opt-In (In)

As a second layer in the namespaceSelector, injection is strictly opt-in. The webhook is only triggered if the namespace carries the label ananse.io/inject In [enabled].

Layer 3: The Go Code Failsafe

The first two layers prevent the deadlock by ensuring the API server never calls a dead webhook. But what if the webhook is up and running, and a control plane pod is created anyway? If the webhook YAML is somehow misconfigured and the API server incorrectly routes a control plane pod to the webhook, the Go code acts as the final bouncer. It guards a different door.
The injectRequired() function contains a hardcoded list of ignored namespaces:

IgnoredNamespaces = []string{  
    metav1.NamespaceSystem, // "kube-system"  
    metav1.NamespacePublic, // "kube-public"  
    "cert-manager",         // Prevent deadlock with cert-manager  
    "ananse-system",        // Don't inject into own control plane  
}

If the request originates from ananse-system, the code explicitly skips the injection and lets the pod boot cleanly without a sidecar.

The probe that dialled itself (kubelet TCP probe on 15006)

One of the most interesting bugs I encountered while building Ananse came from good intentions. In our iptables-init.sh script, we have a safety rule that essentially says: "If a packet is already going to 15006, don't redirect it."
To see how this backfired, let's walk through a standard Kubernetes health check:

• The Probe: Kubelet wants to check if the sidecar is alive, so it sends a TCP probe straight to port 15006.
• The Bypass: The iptables rule catches the request, notices the destination is already 15006, and lets it through without applying a REDIRECT.
• The Handoff: The sidecar picks up the connection. Because it's a proxy, its entire job is to look up the original destination of the request (the SO_ORIGINAL_DST lookup from post 1) and forward the traffic there. But nothing was redirected, so the kernel has no record of an original destination. It just answers with the port the packet arrived on: 15006. So it dutifully dials 127.0.0.1:15006.
• The Loop: That address is itself. The new connection lands on 15006 again, the sidecar repeats the process, and the cycle continues.

Every single health probe started a runaway chain of connections to itself, and a flood of i/o timeouts on every probe interval.

The Fix: The solution was to move the health check off the proxy port and onto a dedicated admin port: 15021.

Because it's an admin port, it handles the request directly. It just answers "I'm alive" and stops right there. It never asks the kernel where the traffic was supposed to go, which means it never dials out. Nothing can loop.

What it costs

With all the benefits a service mesh provides, there is always a trade-off. The true cost of running a mesh in production ranges from added latency to extra CPU and memory in every pod.
We will break down three cost categories (Latency, Machine, and Complexity).

1. Latency (The Round Trip)

Without the mesh, a direct request from Service A to Service B takes only one hop. With Ananse, that path looks like this: Service A -> A's sidecar -> B's sidecar -> Service B
Every request now goes through two proxies. Each hop adds a little time to the request to account for the overhead of accepting the connection, looking up the original destination, dialing it, and copying the bytes over. Ananse was designed for 1 to 3 ms per hop.

2. Machine Resources (CPU and Memory)

Injecting a sidecar means every pod gets an extra container. If you have n injected pods, you are running n additional proxy containers across your cluster.
Over a week of observing everyday production traffic across six services, we saw each sidecar consume roughly 20 to 27 MiB of memory, and under 10 millicores of CPU at its busiest (a hundredth of one core). The busiest sidecars sit in front of the busiest services: CPU follows traffic.
Each sidecar reserves 20 MiB of memory and 10m of CPU, and is capped at 512 MiB and 200m. The memory reservation proved to be a bit too low, as the real baseline sits just above it. Around 32 MiB fits better.
For stability, we ran a k6 soak against v0.3.3 on the dev cluster before go-live, with 50 simulated users going through the gateway the same way real users do: 2,634,971 requests over 8 hours 20 minutes, with 0 pod restarts and 0 sidecar crashes. That run proved the sidecar holds up.

3. Complexity (The Hidden Cost)

Adding a mesh means adding new moving parts to maintain and new ways for the system to break.
You've just read two of them. A webhook that blocked its own control plane from starting, and a health check that flooded the sidecar with connections to itself. Neither bug could exist without a mesh. Both took real debugging to find, because the failure showed up far from the cause.
All of this is to say that a mesh brings its own operational baggage.

What I'd do next

Ananse was originally born just to scratch a curiosity itch. Naturally, I learned a lot of hard lessons along the way. Mistakes were made, quick fixes were applied, and we deliberately chose architectures that optimized for learning rather than perfection, which meant the project took much longer than expected.

Moving forward, here is how we are evolving the mesh:

1. Measure the hop. The plan is rigorous benchmarking, starting with a baseline: the same test with the mesh on and off. After that, every new feature gets benchmarked before it gets merged. The performance delta between the baseline and the pull request becomes the official "cost" of that feature, giving us hard metrics for Ananse's overhead.

2. Replace REDIRECT with TPROXY (if the numbers justify it). Today, every connection gets an address rewrite and a conntrack entry, forcing the sidecar to explicitly ask the kernel where the packet was really going. TPROXY hands the packet over with its original address intact, nothing to rewrite, nothing to look up. It requires a more complex setup (involving policy routing and a transparent socket), so tying back to point #1, we will only implement it if the benchmarks show the conntrack cost is more than we can live with. Still, I believe it is a step in the right direction.

3. Take the proxy out of the pod. Forcing every pod to pay a 20 to 27 MiB "sidecar tax" adds up fast at scale. The longer-term direction is a per-node proxy, or pushing interception down into the kernel using eBPF, similar to how Cilium and Istio's Ambient mode handle it. Ananse needs to feel the ambience too.

Both bugs I shared in this post came from the mesh assuming something about the traffic that turned out not to be true: assuming its own control plane could seamlessly pass through its own webhook, and assuming everything arriving on 15006 was put there by a redirect.
If you've ever run a service mesh in production, I'd love to hear your war stories. Which hidden assumption finally bit you?