Kubernetes networking has historically been split across two layers: the Container Network Interface (CNI), which handles pod-to-pod connectivity and network policy, and the service mesh, which adds application-layer features like mutual TLS, traffic routing, and observability.
For years the common architecture looked like this:
- A CNI plugin such as Calico provided basic network connectivity and Layer 3/4 policy.
- A service mesh like Istio added Layer 7 features using sidecar proxies injected into every pod.
The rise of eBPF-based networking has started to collapse these layers. Modern CNIs such as Cilium — and Calico's newer eBPF dataplane — can enforce policy, capture telemetry, and perform traffic management directly in the Linux kernel without sidecar proxies.
That shift raises a new architectural question for platform teams: if the network layer can already provide identity, encryption, and observability, do you still need a service mesh at all?
Most teams add a service mesh because someone said they needed one. Istio gets installed, sidecars get injected, and six months later the platform team is debugging mTLS failures they didn't have before. The cluster is more observable — and significantly more fragile.
The question worth asking in 2026 isn't "which service mesh should we run?" It's "do we actually need a service mesh at all?"
eBPF changed the answer. If you're working through the K8s Day 2 Method diagnostic framework, this is where the network loop gets architectural — not just operational.
What a Service Mesh Actually Solves
A service mesh exists to solve four problems: mutual TLS between services, traffic management (retries, circuit breaking, weighted routing), Layer 7 observability, and policy enforcement at the application layer.
These are real problems. In a microservices platform with 50+ services, you need mTLS, you need visibility into when a downstream service is degrading, and you need to enforce who can talk to what. The question is whether a sidecar proxy injected into every pod is the right mechanism — or whether the kernel can do it better.
The full container security architecture that frames these requirements — including zero-trust network policy at the cluster level — is covered in the Kubernetes Cluster Orchestration pillar.
The Sidecar Tax
Traditional service meshes like Istio inject an Envoy proxy sidecar alongside every application container. Every network call — even pod-to-pod on the same node — transits through two proxies before reaching its destination. That's latency on every request, plus memory overhead per pod that compounds at scale.
On a cluster with 500 services, the sidecar model can consume 25–50GB of additional memory compared to a sidecar-free alternative. Those aren't abstract numbers — that's real node capacity you're purchasing back.
There's also the operational surface area. Sidecar injection configuration, proxy version alignment, mTLS certificate rotation, Envoy filter ordering — each is a category of failure mode your platform team now owns permanently. The same misconfiguration drift that silently breaks mTLS is the same class of problem covered in the Infrastructure Drift Detection Guide — manual overrides that never get reconciled back to policy.
What eBPF Actually Changes
eBPF lets you attach programs directly to the Linux kernel's network stack. Instead of traffic routing through a userspace proxy, policy enforcement and observability happen at the kernel level — before the standard network stack processes the packet.
For Kubernetes networking this means two things. First, packet processing is faster because the userspace round-trip is eliminated entirely. Second, you can enforce Layer 7 policies — HTTP path filtering, gRPC service filtering, DNS-aware controls — without a sidecar in sight.
Cilium is the most mature implementation of this model. Calico, historically the default CNI for enterprise clusters, now offers an eBPF dataplane alongside its traditional iptables model — though it remains multi-dataplane, falling back to iptables where kernel requirements aren't met. The MTU and overlay encapsulation issues that cause 502 errors in ingress paths apply equally to eBPF tunnel configurations — the debugging methodology in It's Not DNS (It's MTU): Debugging Kubernetes Ingress applies directly here. Need a cluster to test these configurations? DigitalOcean Kubernetes supports Cilium natively on managed clusters.
Calico vs Cilium: The Real Architectural Split
Both tools solve Kubernetes networking. The split is architectural, not just feature-level.
Calico is multi-dataplane — eBPF, iptables, Windows HNS, and VPP are all supported options. Its BGP-based routing model integrates cleanly with physical network infrastructure, which matters in enterprise environments where your K8s cluster doesn't live in isolation. It's predictable, debuggable with standard Linux tools, and reliable in brownfield environments where you can't guarantee a modern kernel on every node.
Cilium is eBPF-native — every feature runs through the kernel. The performance ceiling is higher, the observability model via Hubble is deeper, and L7 policy enforcement is first-class. The major cloud providers have already voted: GKE Dataplane V2, AKS with Azure CNI Powered by Cilium, and AWS EKS increasingly default to Cilium in greenfield configurations. The tradeoff is that eBPF debugging is harder — when a packet drops inside a kernel program, standard Linux tools won't show you why.
CNI DECISION MATRIX
| Scenario | Recommendation | Reasoning |
|---|---|---|
| Greenfield cluster, modern Linux kernel (5.10+) | Cilium | eBPF-native, sidecar-free mesh, Hubble observability out of the box |
| GKE, AKS, or EKS managed cluster | Cilium (default) | Cloud providers have standardised — work with the grain, not against it |
| Brownfield enterprise, mixed OS, BGP fabric integration | Calico | Multi-dataplane flexibility, Windows node support, standard Linux debugging |
| Regulated environment, strict audit-ready compliance | Calico Enterprise | GlobalNetworkPolicy, HostEndpoint protection, compliance-ready controls |
| 500+ services, high throughput, L7 policy required | Cilium | O(1) rule lookup, identity-based policy, no iptables chain bloat at scale |
| Calico running fine, no L7 requirement | Stay put | CNI migration risks cluster-wide outage — don't migrate for its own sake |
The Istio Question
If you're running Istio today, you have options that didn't exist two years ago. Istio's ambient mesh mode — which removes per-pod sidecars in favour of a node-level proxy — reached production readiness in 2025. It's a meaningful improvement to the operational model and brings resource overhead far closer to the eBPF range.
But if you're evaluating a service mesh from scratch and your CNI is Cilium, the answer is that you may not need Istio at all. Cilium covers mTLS via SPIFFE identity, L7 traffic policy, load balancing, and observability through Hubble — without adding a second control plane to manage. The complexity budget you'd spend on Istio is better invested in the application layer.
The caveat: Cilium's mutual authentication model uses eventual consistency for policy sync. In environments where security policy must be instantaneous and fully auditable, Istio's synchronous model remains the more defensible architectural choice.
Migration Reality
One detail glossed over in every CNI comparison: migrating between CNIs in a running cluster requires draining every node. That means a full rolling restart — pod by pod, node by node. In a production cluster with stateful workloads, this is not a Saturday morning task. It's a planned maintenance window with tested rollback procedures.
If you're on Calico and it's working, the bar for switching to Cilium should be "we need what Cilium provides" — not "Cilium is newer." Evaluate against your actual L7 policy requirements, your observability gaps, and whether your node fleet kernel versions support eBPF reliably before committing.
The IaC angle matters here too. Cilium and Calico both have Terraform providers, but provider feature parity lags significantly behind CLI capabilities — especially for Cilium's newer eBPF-specific features. Validate provider support before building your pipeline around capabilities that may not be in the provider yet. The Modern Infrastructure & IaC Learning Path covers the governance framework that prevents policy drift between dev, staging, and production CNI configurations across the full IaC lifecycle.
Architect's Verdict
The service mesh conversation has shifted. The question is no longer which sidecar proxy to run — it's whether the kernel can replace the proxy entirely.
For greenfield platforms on modern infrastructure, Cilium answers that question convincingly. For brownfield enterprise environments with BGP fabric integration, mixed OS node fleets, or compliance requirements that need audit-ready policy controls, Calico remains the more pragmatic choice.
Pick the tool that matches your actual operational reality. Not the one with the best conference talk.
Want the Full Depth?
This post is part of the Kubernetes Day-2 Operations series on Rack2Cloud — field-tested architecture guidance for platform engineers.
The full version includes the complete CNI decision framework, IaC governance patterns for network policy drift, and the Terraform Feature Lag Tracker for validating CNI provider support before you build your pipeline.
→ Read the full post on Rack2Cloud
If you found this useful, the K8s Day-2 series covers the full diagnostic framework — Identity Loop, Compute Loop, Network Loop, and Storage Loop.
Originally published at rack2cloud.com.
Top comments (0)