This is Level 8 of our Kubernetes networking series, and it's where we finally go inside the Service dataplane. Back in Level 3, we learned that a client Pod talks to a Service's ClusterIP, which somehow routes traffic to a backend Pod:
Client Pod → Service (10.96.120.50:80) → Backend Pod (10.244.1.10:80)
We deliberately left one question unanswered until now: how does traffic sent to that Service IP actually end up at a real Pod? That's exactly what this article unpacks.
Table of Contents
- The Problem: A Service IP Isn't a Real Interface
- The Basic Idea: A Service Dataplane
- What Is kube-proxy?
- kube-proxy's Job
- kube-proxy Doesn't Create Pods
- kube-proxy Runs on Every Node
- The Classic Mental Model: iptables and DNAT
- What Is DNAT?
- Tracing the Packet Journey
- The Full Picture: DNS to Pod
- iptables in Kubernetes
- Control Plane vs Dataplane
- Why EndpointSlices Matter Here
- Load Balancing Across Endpoints
- Session Affinity
- The Scaling Problem with iptables
- IPVS: IP Virtual Server
- iptables vs IPVS
- An Important Modern Nuance
- eBPF Enters the Picture
- Why eBPF Is Interesting for Kubernetes
- Cilium and eBPF
- eBPF Can Replace Parts of kube-proxy
- Don't Confuse kube-proxy Replacement with CNI
- Three Generations to Remember
- What kube-proxy Actually Watches
- Why Services Survive Pod Churn
- Where CNI Fits Into All of This
- Service vs CNI vs kube-proxy: A Reference Table
- Following One Complete Packet
- What Is SNAT?
- Why SNAT Is Needed
- DNAT vs SNAT
- Why Return Traffic Works
- NodePort, Revisited
- LoadBalancer, Revisited
- A Layered Troubleshooting Model
- A Production Troubleshooting Tree
- What to Inspect on a Node
- How Do I Know Which Dataplane I'm Using?
- The Most Important Conceptual Separation
- Interview Questions
- Level 8 Mental Model
- What's Next: Kubernetes Networking Internals
The Problem: A Service IP Isn't a Real Interface
Suppose we have a frontend Pod at 10.244.1.5, and a Service named backend with ClusterIP 10.96.120.50:80, backed by three Pods:
10.244.1.10:80
10.244.2.10:80
10.244.3.10:80
The frontend runs curl http://backend:80. DNS resolves backend → 10.96.120.50, so the outgoing packet looks like:
SOURCE: 10.244.1.5
DESTINATION: 10.96.120.50:80
Here's the strange part: 10.96.120.50 usually isn't a real, physical network interface attached to any Pod. So who transforms 10.96.120.50:80 into something like 10.244.2.10:80? That transformation is the job of the Service dataplane — and that's exactly what this article is about.
The Basic Idea: A Service Dataplane
Service (10.96.120.50:80)
↓
Service dataplane
↓
Backend Pod (10.244.2.10:80)
Historically, Kubernetes clusters implemented this using kube-proxy. Modern clusters can implement it very differently. The core concepts we need to understand across this article are: kube-proxy, iptables, IPVS, and eBPF.
What Is kube-proxy?
Despite its name, kube-proxy isn't simply a traditional HTTP proxy — that's a genuinely useful interview point to remember. kube-proxy is a component that helps implement Kubernetes Service networking on each Node:
Service
↓
kube-proxy / service dataplane
↓
backend Pods
It watches Kubernetes objects — specifically Services and EndpointSlices — and programs the Node's networking rules/dataplane to match what it observes.
kube-proxy's Job
Given a Service backend at 10.96.120.50:80 with EndpointSlice entries pointing to 10.244.1.10:80, 10.244.2.10:80, and 10.244.3.10:80, kube-proxy's conceptual job is simple:
"When traffic arrives for this Service IP and port, send it to one of these backend endpoints."
10.96.120.50:80
↓
┌──────┼──────┐
↓ ↓ ↓
Pod 1 Pod 2 Pod 3
kube-proxy Doesn't Create Pods
Another important distinction worth locking in:
Deployment → creates/manages Pods
Service → defines a stable endpoint
kube-proxy/dataplane → implements Service traffic forwarding
Three genuinely different responsibilities, each handled by a different component.
kube-proxy Runs on Every Node
kube-proxy typically runs as a DaemonSet, so every Node gets its own instance:
Node 1 → Pods + kube-proxy
Node 2 → Pods + kube-proxy
Node 3 → Pods + kube-proxy
Each Node needs the ability to handle Service traffic that arrives locally.
The Classic Mental Model: iptables and DNAT
Historically, kube-proxy commonly programmed iptables rules to do this work:
Pod
↓
Service IP
↓
iptables rules
↓
DNAT
↓
Pod IP
The key term here is DNAT.
What Is DNAT?
DNAT = Destination Network Address Translation. Suppose a packet starts out as:
SOURCE: 10.244.1.5
DESTINATION: 10.96.120.50:80
The Service dataplane can rewrite the destination:
SOURCE: 10.244.1.5
DESTINATION: 10.244.2.10:80
So 10.96.120.50 → 10.244.2.10 — that transformation is destination NAT in action.
Tracing the Packet Journey
Let's slow this down and walk through it explicitly.
Frontend: 10.244.1.5 — Service: 10.96.120.50:80 — Backend: 10.244.2.10:80
- The frontend sends a packet:
SRC = 10.244.1.5,DST = 10.96.120.50:80. - The Node's Service dataplane recognizes
10.96.120.50:80and selects a backend, say10.244.2.10:80. - The packet effectively becomes:
SRC = 10.244.1.5,DST = 10.244.2.10:80. - From here, normal Pod networking (CNI, covered in Level 7) takes over and delivers it.
The Full Picture: DNS to Pod
Connecting this back to Level 4:
Pod
↓
DNS
↓
Service name
↓
ClusterIP
↓
Service dataplane
↓
Endpoint
↓
Pod IP
↓
CNI networking
↓
destination Pod
Or, with concrete values:
frontend (10.244.1.5)
| "backend"
↓
DNS
↓
10.96.120.50
↓
Service dataplane
↓
10.244.2.10
↓
CNI
↓
backend Pod
iptables in Kubernetes
Linux has a built-in packet-filtering and NAT framework called iptables. Kubernetes can program iptables rules so that Service traffic gets redirected to the correct backend endpoint:
Packet
↓
iptables
↓
Service rule
↓
Backend selection
↓
DNAT
↓
Pod
Conceptually, given a Service at 10.96.120.50:80 with endpoints 10.244.1.10:80, 10.244.2.10:80, and 10.244.3.10:80, the Node might carry rules that resemble: "traffic to 10.96.120.50:80 → choose an endpoint → DNAT to one of the three addresses above." Real Kubernetes iptables rule sets are considerably more complex than this simplification — don't worry about memorizing actual chain names yet.
Control Plane vs Dataplane
It helps to think of Kubernetes networking in two layers:
Control plane decides what should happen: which Pods exist, which Services exist, which endpoints belong to a Service.
Dataplane actually handles packets.
Control plane → "Service backend Pods are A, B, C"
↓
Dataplane → "Send packets to A/B/C"
Why EndpointSlices Matter Here
Recall from Level 3: a Service like backend has an associated EndpointSlice listing its live backend IPs. The Service dataplane depends directly on this information:
Service + EndpointSlice → Service dataplane → forwarding rules
Without accurate, up-to-date EndpointSlice data, the dataplane has nothing correct to forward traffic to.
Load Balancing Across Endpoints
Given a Service with three backend Pods, a connection to the Service might land on any one of them:
Request 1 → Pod A
Request 2 → Pod C
Request 3 → Pod B
Request 4 → Pod A
The exact algorithm and distribution behavior depends on the dataplane implementation and configuration. The key idea to hold onto: the Service is a stable front door, while the dataplane is what actually selects a backend endpoint for each connection.
Session Affinity
Kubernetes Services can optionally enable session affinity, e.g.:
sessionAffinity: ClientIP
With this enabled, the same client tends to keep landing on the same backend Pod:
Client A → Pod A
Client A → Pod A
Client A → Pod A
Again, the precise implementation details depend on which dataplane your cluster is running.
The Scaling Problem with iptables
The classic kube-proxy → iptables → DNAT → Pod flow works well for many clusters. But there's a scaling consideration: compare 10 Services with 100 total endpoints to 10,000 Services with 100,000 total endpoints. The volume of networking rules involved can grow enormous. This is exactly the pressure that led to IPVS, and eventually to eBPF.
IPVS: IP Virtual Server
IPVS stands for IP Virtual Server — a Linux kernel technology purpose-built for high-performance load balancing:
Service IP (10.96.120.50:80)
↓
IPVS
↓
┌───┼───┐
↓ ↓ ↓
Pod A Pod B Pod C
Instead of thinking in terms of a long chain of packet-filter rules, think of IPVS as a kernel-level virtual server / load-balancing mechanism purpose-built for exactly this job.
iptables vs IPVS
Simplified comparison:
iptables: packet → rules → rules → rules → backend
IPVS: packet → virtual service → backend selection
IPVS can offer more specialized load-balancing behavior, and historically was attractive for clusters with large numbers of Services thanks to its performance characteristics. That said, modern Kubernetes networking has moved beyond simply choosing between these two — eBPF-based dataplanes are increasingly significant.
An Important Modern Nuance
You may hear the claim: "IPVS is always faster than iptables." Don't treat that as an absolute rule. Real-world performance depends on cluster size, traffic patterns, kernel version, configuration, the specific dataplane implementation, and even the underlying hardware. The important conceptual difference to remember is simpler:
iptables → packet filtering/NAT rules
IPVS → kernel virtual-server load balancing
eBPF Enters the Picture
Now we reach a genuinely important modern technology: eBPF. You don't need to master every detail yet — at a high level:
eBPF = programmable logic that can run inside the Linux kernel.
This lets networking systems perform operations directly in the kernel, rather than bouncing through separate userspace components or long rule chains.
Why eBPF Is Interesting for Kubernetes
The traditional path looks like:
Packet → iptables rules → NAT → routing
An eBPF-based networking implementation can inject custom packet-processing logic straight into the kernel:
Packet → eBPF program → routing / load balancing / policy → destination
This can avoid some of the overhead inherent in traditional networking paths and enables much more flexible dataplane behavior.
Cilium and eBPF
Cilium is one of the best-known Kubernetes networking implementations built on eBPF:
Kubernetes → Cilium → eBPF → Linux kernel → networking + load balancing + NetworkPolicy + observability
This is exactly why Cilium came up briefly back in Level 7 — and now the pieces should be clicking into place.
eBPF Can Replace Parts of kube-proxy
This is an important, genuinely modern Kubernetes concept. Some eBPF-based networking implementations can provide Service load balancing entirely on their own. Instead of:
Service → kube-proxy → iptables → Pod
you can have:
Service → eBPF dataplane → Pod
In such a setup, kube-proxy may not even be needed for Service handling at all. This is commonly referred to as "kube-proxy replacement," and Cilium is a well-known example of a system that offers it.
Don't Confuse kube-proxy Replacement with CNI
Keep these terms distinct in your head:
CNI = networking interface/ecosystem
Cilium = a networking implementation
eBPF = a kernel technology
kube-proxy replacement = one specific capability/configuration
They're related, but not interchangeable. A cleaner way to see the relationship:
Cilium → uses eBPF → can implement Kubernetes networking → can provide Service load balancing → can replace kube-proxy
Three Generations to Remember
A simple learning model for Service traffic handling:
Service traffic
Service IP
|
┌──────────────┼──────────────┐
↓ ↓ ↓
iptables IPVS eBPF
↓ ↓ ↓
kernel rules virtual server kernel programs
↓ ↓ ↓
Pod
This isn't a strict historical replacement chain — real clusters can (and do) use any of these approaches — but it's a useful conceptual comparison to keep handy.
What kube-proxy Actually Watches
kube-proxy needs live information about Services and EndpointSlices. Whenever something changes — a Pod is added or removed, a Service is updated, an endpoint changes — the Service dataplane needs to be kept in sync:
EndpointSlice changes
↓
kube-proxy/dataplane notices
↓
forwarding state updated
↓
new traffic uses current endpoints
This is precisely how Kubernetes handles the constant churn of dynamic Pods without breaking client connectivity.
Why Services Survive Pod Churn
Suppose Service backend (10.96.120.50) initially has Pod A (10.244.1.10) and Pod B (10.244.2.10) as backends. Pod A dies, and a new Pod C (10.244.3.10) appears. The EndpointSlice updates:
OLD: A, B
NEW: B, C
But the Service IP itself never changes — it stays 10.96.120.50. The dataplane simply updates its internal backend information, and clients keep using backend without ever needing to know that a Pod was replaced underneath them. This is one of the single most powerful abstractions in all of Kubernetes.
Where CNI Fits Into All of This
Combining Level 7 and Level 8: CNI provides the underlying Pod network:
Pod → eth0 → veth → Node → Pod network
The Service dataplane provides Service-level forwarding:
Service IP → backend Pod
Together:
Kubernetes networking
Pod
| CNI
↓
Pod network
↓
Service dataplane
↓
another Pod
The exact ordering within the Linux packet path can get more nuanced in practice, but this separation of responsibilities is the key mental model to carry forward.
Service vs CNI vs kube-proxy: A Reference Table
| Component | Main Job |
|---|---|
| CNI/network implementation | Pod networking |
| IPAM | Pod IP allocation |
| Service | Stable virtual endpoint |
| EndpointSlice | Backend endpoint information |
| kube-proxy | Traditional Service dataplane implementation |
| iptables | Linux packet filtering/NAT framework |
| IPVS | Linux virtual-server load balancing |
| eBPF | Programmable kernel dataplane technology |
Following One Complete Packet
Let's trace one full request end-to-end. We have frontend (10.244.1.5), Service backend (10.96.120.50:80), and a backend Pod (10.244.2.10:80). The frontend runs curl http://backend.
Step 1 — DNS: backend → 10.96.120.50.
Step 2 — Application sends packet: SRC = 10.244.1.5, DST = 10.96.120.50:80.
Step 3 — Service dataplane: either kube-proxy → iptables/IPVS, or an eBPF dataplane, takes over.
Step 4 — Destination translation/selection: 10.96.120.50:80 → 10.244.2.10:80.
Step 5 — Pod networking: CNI provides the actual route to 10.244.2.10.
Step 6 — Backend receives traffic: Frontend → Service → Service dataplane → Backend Pod.
What Is SNAT?
We've covered DNAT — now meet its counterpart: SNAT = Source Network Address Translation.
DNAT → destination changes
SNAT → source changes
For example, SRC = Pod IP, DST = external destination might become SRC = Node IP, DST = external destination. This matters specifically when Pods communicate outside the cluster.
Why SNAT Is Needed
Suppose Pod 10.244.1.10 tries to reach the internet. The external destination generally has no idea how to route return traffic back to 10.244.1.10 — that's a private, cluster-internal address. So the networking system may perform SNAT:
10.244.1.10
↓ SNAT
192.168.1.10 (Node IP)
Conceptually: Pod → SNAT → Node IP → Internet. The exact behavior depends on your specific cluster's networking architecture.
DNAT vs SNAT
Worth memorizing permanently:
DNAT — D = Destination
SNAT — S = Source
Example (Service → Pod): 10.96.120.50 → DNAT → 10.244.2.10
Example (Pod → outside): 10.244.1.10 → SNAT → Node IP
Why Return Traffic Works
This is where NAT/connection tracking becomes essential. Given Client (10.244.1.5) → Service (10.96.120.50) → Backend (10.244.2.10), the forward path goes Client → Service IP → DNAT → Backend. For the response to correctly find its way back, the networking stack tracks the connection state so return traffic can be translated appropriately in reverse:
Forward: Service → Backend
Return: Backend → Client
This bookkeeping is handled through connection/NAT state tracking within the Linux networking stack (often referred to as "conntrack").
NodePort, Revisited
With a NodePort Service, given a Node IP of 192.168.1.10 and NodePort 30080, external traffic to 192.168.1.10:30080 can be forwarded toward the Service's backend Pods:
External client
↓
NodeIP:30080
↓
Service dataplane
↓
Backend Pod
The exact mechanics again depend on which dataplane your cluster is running.
LoadBalancer, Revisited
With type: LoadBalancer, the flow typically looks like:
Internet
↓
Cloud Load Balancer
↓
Node / Service
↓
Backend Pods
Both the cloud load balancer and the Kubernetes Service dataplane participate in the overall traffic path — which is exactly why troubleshooting external traffic often requires checking multiple distinct layers.
A Layered Troubleshooting Model
Now that we understand the full stack, we can build a much better troubleshooting model. If curl http://backend fails, don't immediately blame DNS. Break it down layer by layer:
1. DNS
↓
2. Service
↓
3. EndpointSlice
↓
4. Service dataplane
↓
5. Pod networking
↓
6. Application
Step 1 — DNS:
nslookup backend
Does it resolve to the expected Service IP? If backend → 10.96.120.50, DNS is probably fine.
Step 2 — Service:
kubectl get svc backend
Check the ClusterIP, port, and targetPort.
Step 3 — EndpointSlice:
kubectl get endpointslices
kubectl describe svc backend
Ask directly: does the Service actually have backend endpoints? If you see Endpoints: <none>, don't waste time debugging iptables yet — the Service simply has no backends to send traffic to.
Step 4 — Pod:
kubectl get pods -o wide
Confirm the backend Pods are Running, Ready, and have assigned IPs.
Step 5 — Service dataplane: Ask whether the Service dataplane is correctly forwarding traffic — investigate kube-proxy, iptables, IPVS, or your eBPF/CNI dataplane as appropriate. This is exactly where Level 8 knowledge pays off.
Step 6 — CNI: If Service forwarding looks correct but the destination Pod still isn't reachable, investigate CNI, routing, the Pod's interface, Node connectivity, and NetworkPolicy.
A Production Troubleshooting Tree
Request fails
|
↓
DNS?
/ \
NO YES
| |
Fix DNS Service?
/ \
NO YES
| |
Fix Service EndpointSlice?
/ \
NO YES
| |
Fix selectors Dataplane?
/ \
NO YES
| |
kube-proxy/ CNI/
eBPF/etc. routing/
policy
This structured approach beats randomly running commands and hoping something jumps out.
What to Inspect on a Node
For a traditional kube-proxy cluster using iptables, you can inspect:
iptables-save
# or
iptables -t nat -S
These outputs can be extremely large on real clusters — don't be surprised by the volume. For IPVS-based setups, if installed:
ipvsadm -L
For eBPF-based implementations, the relevant tooling depends heavily on the specific implementation — Cilium, for example, ships its own observability and status tooling. Don't run diagnostic commands blindly in production — first figure out which dataplane your cluster actually uses.
How Do I Know Which Dataplane I'm Using?
Start broad:
kubectl get pods -A
Look for names like kube-proxy, calico, cilium, flannel, or aws-node. Then:
kubectl get daemonsets -A
For example, seeing a kube-proxy DaemonSet suggests kube-proxy is deployed — but you should still inspect its configuration to determine whether it's actually running in iptables mode, IPVS mode, or something else entirely.
The Most Important Conceptual Separation
By now, you should be able to cleanly separate these layers in your head:
Kubernetes networking
Service
|
↓
Service dataplane
┌─────────┼─────────┐
↓ ↓ ↓
iptables IPVS eBPF
\ | /
└────────┼────────┘
↓
Pod IP
|
↓
CNI network
|
↓
Pod
Interview Questions
Q1. What does kube-proxy do?
It traditionally implements Kubernetes Service networking on Nodes by programming a Service dataplane.
Q2. Is kube-proxy an HTTP proxy?
No — the name is genuinely misleading.
Q3. What is a ClusterIP?
A stable virtual IP associated with a Kubernetes Service.
Q4. How does a ClusterIP actually reach a Pod?
Through the Service dataplane — traditionally via iptables/IPVS, or through an alternative dataplane such as eBPF.
Q5. What is DNAT?
Destination Network Address Translation — conceptually, Service IP → Pod IP.
Q6. What is SNAT?
Source Network Address Translation — conceptually, Pod IP → Node IP in some outbound scenarios.
Q7. What is IPVS?
Linux IP Virtual Server — a kernel-level load-balancing mechanism.
Q8. What is eBPF?
A Linux kernel technology that allows programmable logic to run in kernel contexts, used by modern networking systems for routing, load balancing, security, and observability.
Q9. Can eBPF replace kube-proxy?
Yes — some implementations, such as Cilium, can provide full Kubernetes Service functionality without needing kube-proxy at all.
Q10. What does EndpointSlice provide?
Information about the current backend endpoints associated with a Service.
Q11. Is CNI the same as kube-proxy?
No. CNI/network implementation handles Pod networking; kube-proxy/service dataplane handles Service forwarding.
Q12. What happens if a Service has no EndpointSlices/endpoints?
The Service has no backend Pods to send traffic to — regardless of whether the Service dataplane itself is working correctly.
Level 8 Mental Model
If you take away only one diagram from this entire article, make it this one:
Service
10.96.120.50:80
|
↓
Service dataplane
|
┌───────────┼───────────┐
↓ ↓ ↓
iptables IPVS eBPF
└───────────┼───────────┘
↓
Pod endpoint
10.244.2.10
|
↓
CNI
|
↓
Pod
And the responsibility of each layer:
- CNI → "How do Pods communicate?"
- Service → "What stable endpoint do clients use?"
- EndpointSlice → "Which Pods are backends?"
- kube-proxy / dataplane → "How do Service packets reach those backends?"
- iptables / IPVS / eBPF → "How is that forwarding actually implemented?"
What's Next: Kubernetes Networking Internals
In Level 9, we'll connect everything into the actual Linux packet path — diving into network namespaces, veth pairs, bridges, routing tables, conntrack, NAT, the kube-proxy/CNI interaction, Node-to-Pod versus Pod-to-Pod traffic, and exactly what happens to a single packet as it moves through a Kubernetes Node.
Conclusion
The Service dataplane is where Kubernetes networking stops being abstract and becomes real packet manipulation. A ClusterIP is never a physical address a Pod actually owns — it's a rule, enforced by kube-proxy (or an eBPF-based alternative), that rewrites destinations via DNAT and tracks connections so return traffic finds its way home. Whether that enforcement happens through iptables rule chains, IPVS's kernel-level virtual server, or a fully custom eBPF dataplane, the underlying job is the same: turn a stable Service IP into real traffic to a real, currently-healthy Pod.
The biggest practical takeaway: when Service traffic breaks, work the layers in order — DNS, Service, EndpointSlice, dataplane, then CNI. An empty EndpointSlice will waste hours of your time if you jump straight to debugging iptables rules instead of checking it first.
Enjoyed this deep dive? Follow along for Level 9, where we trace a single packet's complete journey through the Linux networking stack inside a real Kubernetes Node. Drop your questions in the comments, and share this with a teammate who still thinks kube-proxy is a literal proxy server. Let's keep building this Kubernetes networking roadmap together. 🚀
Top comments (0)