Understanding Kubernetes Datapath With Cilium

Kubernetes network might look simple when a cluster is small.

A pod can communicate with another pod, an application can expose a service, and clients can connect to the application.

However as the cluster grows, networking becomes more than just connecting one pod to another.

For example, consider a simple application running inside Kubernetes. A request may start from one Pod and reach another Pod through a Kubernetes Service.

Now consider a request coming from outside the cluster.

The traffic needs to enter the cluster, reach a LoadBalancer address, be handled by the networking layer, and eventually reach the application Pod.

This raises an important question:

What actually happens to a packet as it travels from one Pod, through a Kubernetes Service, and eventually reaches an application from outside the cluster?

Answering this question requires looking beyond Kubernetes objects such as Pods and SFervices. We also need to understand the components responsible for turning those objects into actual network traffic.

Traditionally, Kubernetes networking relies on multiple components working together to provide service connectivity and traffic forwarding.

In this lab, we explore what changes when Cilium becomes the networking layer.

Rather than treating Cilium simply as a CNI plugin, we will look at how it participates in the Kubernetes networking datapath and how its components work together to handle different types of traffic.

This article will cover several topics:

• Cilium's networking model

• eBPF based service handling

• kube proxy replacement

• LoadBalancer IP allocation with LB-IPAM

• Layer 2 advertisement

• Envoy based ingress

Kubernetes networking is not just about assigning IP addresses to Pods. It is about building a datapath that determines how packets move between workloads, Services, nodes, and external networks.

The Environment

Before looking at the datapath, it is useful to establish the environment in which the experiments are performed.

The Kubernetes cluster runs on Ubuntu virtual machines hosted by a physical server using KVM/libvirt. The cluster is provisioned with kubeadm, while containerd is used as the container runtime.

The environment is intentionally kept close to a small on premises Kubernetes deployment rather than relying on a managed Kubernetes service. This makes the networking behavior, particularly LoadBalancer IP allocation and Layer 2 advertisement, visible from the underlying network.

Kubernetes Cluster

The cluster uses a multi control-plane architecture to provide a highly available Kubernetes control plane.

The Kubernetes API is exposed through HAProxy to provide a stable endpoint for cluster administration and for nodes joining the control plane.

The cluster consists of:

• 3 control plane nodes

• 3 worker nodes

• Ubuntu virtual machines

• kubeadm for cluster provisioning

• containerd as the container runtime

• HAProxy

Networking Environment

The cluster runs inside a libvirt network hosted on a physical server.

At a high level, the environment looks like this:

The Kubernetes nodes communicate through the node network, while Pods use a separate Pod CIDR managed by Cilium.

For the experiments in this article, the relevant network ranges are:

Component	Network
Node network	192.168.122.0/24
Pod CIDR	10.244.0.0/16
LoadBalancer IP range	192.168.122.241–192.168.122.254
Kubernetes	v1.34.11
Cilium	v1.20.2

The LoadBalancer addresses are allocated from the same Layer 2 network as the Kubernetes nodes.

This is important because the LoadBalancer IP is not provided by an external cloud provider. Cilium must allocate an address from the configured pool and make that address reachable from the local network.

Infrastructure Automation

The underlying Kubernetes environment is provisioned and configured through Ansible.

The automation covers the infrastructure and cluster setup, including the Kubernetes nodes, container runtime, control-plane configuration, HAProxy, and Cilium deployment.

The implementation is maintained in the accompanying GitHub repository:

https://github.com/muhammadyulasfipahrizal/cilium-k8s.git

The repository is used to make the underlying Kubernetes environment reproducible, while this article focuses on understanding the networking behavior implemented on top of it.

Kubernetes Networking Before Cilium

Before looking at what Cilium changes, we need to establish the networking model that Kubernetes presents to applications.

A Kubernetes application does not normally communicate using a single type of IP address.

A Pod has its own IP address. A Service provides a stable virtual address in front of a group of Pods. Nodes have their own addresses on the underlying network, and applications exposed outside the cluster may use a LoadBalancer address or an Ingress endpoint.

These addresses belong to different layers of the networking path.

Different Networking Layers

A simplified view of the path looks like this:

Pod network

Each Pod receives an IP address from the cluster's Pod CIDR. This address identifies the Pod within the Kubernetes networking environment.

For example, an application might have an address such as:

10.244.x.x

The important property is that the application does not need to know which physical or virtual machine is hosting the Pod. The networking layer is responsible for making Pod connectivity work across nodes.

Service network

A Service provides a stable virtual IP in front of one or more Pods. Instead of connecting directly to a changing Pod IP, clients can connect to the Service:

The Service therefore introduces an abstraction between the client and the actual workload.

The client does not need to know which Pod should receive the packet.

Something inside the networking datapath must make that decision.

A Service definition describes the desired connectivity:

Service
  name: nginx
  port: 80

But a packet arriving at the Service IP does not automatically know that it should be forwarded to one of the backend Pods.

The networking implementation must translate that logical Service into forwarding behavior.

Node network

Every Kubernetes node has an IP address on the underlying network. This address allows the node to communicate with other nodes and with networks outside the cluster.

At this point, the packet is no longer concerned only with Kubernetes abstractions. It has to interact with the actual network connecting the nodes.

External network

For externally exposed applications, Kubernetes can provide an address that is reachable outside the cluster. Depending on the environment, this may be a cloud provided LoadBalancer address, a manually assigned address, or an address advertised directly on the local network.

In this lab, the LoadBalancer addresses come from the local node network.

This creates a complete path across several networking layers:

Where kube-proxy Fits

One of the key components in the traditional Kubernetes networking model is kube proxy.

A Kubernetes Service can have multiple backend endpoints, while clients see only one stable Service IP.

The networking layer therefore needs to perform a service translation:

The exact implementation can vary, but the fundamental idea remains the same:

Kubernetes needs a datapath mechanism that turns Service definitions and endpoints into packet forwarding rules.

When a request is sent to a Service, kube proxy is responsible for programming the node's networking layer so that traffic can be redirected

Cilium's Role

Pods need connectivity, Services need to map stable virtual addresses to backend workloads, and externally exposed applications need a path between the Kubernetes network and the surrounding network.

Traditionally, these responsibilities are implemented by multiple networking components.

Cilium approaches the problem differently.

It starts as a CNI plugin, but its role does not end after a Pod receives an IP address. Cilium provides the datapath responsible for processing network traffic and can take responsibility for Service handling, network policy, load balancing, observability, and other networking functions.

CNI vs Networking Datapath

The Container Network Interface (CNI) defines how a container runtime invokes networking plugins when a Pod is created or removed.

This is an important responsibility, but it represents only one part of Kubernetes networking.

Once the Pod is running, packets continuously move through the networking datapath.

The CNI lifecycle answers questions such as:

How does this Pod become connected to the network?

The datapath answers a different question:

What should happen to each packet after the Pod is connected?

Cilium participates in both.

Cilium can provide the initial Pod networking, while its agents and datapath continue to manage how traffic is processed after the Pod has started.

This also allows several Kubernetes networking functions to be handled by the same networking system.

The result is a networking layer that understands more than just Pod interfaces.

It can observe Kubernetes resources and translate their desired state into actual packet processing behavior.

eBPF as the Datapath

Cilium uses Extended Berkeley Packet Filter (eBPF) to implement significant parts of its networking datapath.

eBPF allows programs to run inside the Linux kernel at specific networking and system execution points.

For Cilium, this provides a way to process packets close to where they enter and move through the kernel networking stack.

The high-level architecture can be simplified to:

The Cilium Agent runs in userspace and is responsible for observing the Kubernetes environment and configuring the datapath.

The eBPF programs then perform packet processing in the kernel.

The agent does not need to make every packet decision itself.

Instead, it configures the datapath so that packets can be processed directly in the kernel.

This becomes particularly important for Kubernetes Services.

With a traditional kube-proxy based architecture, Service definitions are translated into networking rules such as iptables or IPVS rules.

With Cilium, Service handling can instead be implemented using eBPF-based load-balancing and forwarding.

compared with:

The important difference is not simply that one uses iptables while the other uses eBPF.

The larger architectural difference is where networking decisions are implemented and how the datapath is programmed.

Cilium moves important packet-processing logic into the kernel, allowing the datapath to make decisions without requiring a userspace proxy to process every connection.

However, this does not mean that everything in Cilium runs inside the kernel.

Cilium still has userspace components.

For example, the Cilium Agent manages the datapath, while Envoy can handle application-layer traffic such as HTTP and TLS when Cilium's ingress functionality is used.

This distinction will become important later.

Not every networking problem should be solved at the same layer.

Some decisions can be made at the kernel level, such as Service load balancing and packet forwarding.

Other decisions require understanding application-level protocols, where a proxy such as Envoy becomes useful.

This gives us a more useful mental model of Cilium:

Cilium is not only responsible for connecting Pods to the network. It provides a programmable networking datapath that connects Kubernetes state with actual packet processing.

From here, the individual components we enabled in the lab start to make more sense.

1. Kube-proxy replacement, changes how Kubernetes Services are implemented.

2. LB-IPAM, determines how LoadBalancer addresses are allocated.

3. L2 Announcement, makes those addresses reachable from the local network.

4. Envoy, provides the application-layer entry point for Ingress traffic.

These are not isolated features. They are different parts of the same networking path.

Establishing the Baseline

Before changing the Kubernetes networking datapath, we first need to establish what the cluster looks like in its initial state.

Verify the Cilium Nodes

Cilium maintains its own representation of Kubernetes nodes through CiliumNode resources.

We can inspect them with:

kubectl get ciliumnodes

What this demonstrates:

• Cilium recognizes the Kubernetes nodes.

• Each node participates in the Cilium networking environment.

• Kubernetes node networking state is represented through Cilium resources.

This gives us the first indication that Cilium is not simply providing connectivity during Pod creation.

It maintains networking state for the cluster and uses that state to construct its datapath.

---

Pod-to-Pod Communication

The next step is to verify basic workload connectivity.

We create two lightweight test Pods and use them to observe how traffic moves through the cluster.

The simplified communication path is:

We first verify that a Pod can resolve a Kubernetes Service:

nslookup kubernetes.default.svc.cluster.local

What this demonstrates:

• The Pod can reach Kubernetes DNS.

• Kubernetes Service discovery is functioning.

• The Pod network can communicate with the cluster's DNS service.

We can then verify direct connectivity between the test Pods.

Here i create two pods using busybox:1.36 and testing their connectivity inside the cluster

What this demonstrates:

• Pods can communicate across the Kubernetes network layer.

• The CNI datapath is forwarding workload traffic successfully.

• Pod connectivity works before introducing the additional Cilium features examined later.

Replacing kube-proxy with Cilium

The baseline confirms that the cluster can provide basic Pod networking and Kubernetes Service connectivity.

However, there is still another component involved in the Service datapath: kube-proxy.

In a traditional Kubernetes deployment, kube-proxy watches Services and their endpoints and programs the node networking layer so that traffic sent to a Service can reach one of its backend Pods.

This model works, but it introduces another component responsible for translating Kubernetes Service state into packet forwarding behavior.

Since Cilium already provides a programmable networking datapath, a natural question is:

What happens if Cilium takes responsibility for Kubernetes Service handling as well?

Why Replace kube-proxy?

With the traditional Kubernetes networking model, kube-proxy programs the node's networking rules to implement Service forwarding.

With Cilium's kube-proxy replacement, these Service networking functions can instead be handled directly by Cilium and eBPF, allowing traffic to be processed closer to the kernel networking layer.

Instead of:

we move toward a datapath where Cilium handles Service translation:

This changes the architecture rather than simply changing a configuration flag.

The main benefits are:

• A more direct datapath, Service translation and forwarding can be handled by eBPF in the kernel instead of relying on kube-proxy-managed networking rules.

• Lower networking overhead, Removing an additional Service-proxying layer can simplify packet processing, particularly in clusters with a large number of Services and endpoints.

• Better integration with Cilium, Service handling becomes part of the same datapath used for networking, load balancing, security, and observability.

• Scalable load balancing, Cilium can perform Service load balancing using eBPF-based mechanisms, including efficient handling of large endpoint sets.

• More control over networking behavior, Cilium exposes features such as socket-level load balancing, direct routing, and other eBPF-based datapath capabilities that go beyond traditional kube-proxy behavior.

---

Enabling kube-proxy Replacement

Cilium exposes kube-proxy replacement through its configuration.

The relevant setting in this lab is:

kubeProxyReplacement=true

The configuration tells Cilium to take responsibility for Kubernetes Service handling through its own datapath.

The Cilium Agent observes Kubernetes Service and endpoint state and programs the eBPF datapath accordingly.

This means that the Service abstraction remains the same from the application's perspective.

Applications still communicate with a Kubernetes Service.

What changes is the mechanism underneath that abstraction.

---

Removing kube-proxy

Once Cilium's kube-proxy replacement is active, we can inspect the existing kube-proxy DaemonSet:

kubectl -n kube-system get ds kube-proxy

The DaemonSet can then be removed from the cluster.

kubectl -n kube-system delete ds kube-proxy

If Cilium is correctly providing the Service datapath, removing kube-proxy should not prevent applications from communicating through Kubernetes Services.

---

Verification

Now we repeat the same networking tests used during the baseline.

This is important because we want to compare behavior rather than introduce an entirely different test.

The tests include:

• DNS resolution

• Pod-to-Pod connectivity

• Kubernetes Service connectivity

First, we verify that Kubernetes DNS is still reachable.

We then repeat the Pod-to-Pod connectivity test.

Finally, we test connectivity through a Kubernetes Service.

The important observation is that the Kubernetes abstraction has not changed. What changed is the datapath underneath it.

From ClusterIP to LoadBalancer

A Pod can communicate with another Pod, and a Kubernetes Service can provide a stable endpoint in front of those workloads.

But what happens when the client is outside the cluster?

A LoadBalancer Service introduces another requirement. The Service needs an IP address that external clients can actually reach.

In a cloud environment, this is usually handled by a cloud controller that provisions an external load balancer and assigns an address to the Service.

Our environment does not have a cloud provider.

Instead, we want Kubernetes to expose a Service using an address from the same Layer 2 network as the Kubernetes nodes:

This creates two separate networking problems:

1. Which IP address should the LoadBalancer Service use?

2. How does the local network learn that this IP is reachable through the Kubernetes cluster?

Cilium solves these problems with two different components:

LB-IPAM and L2 Announcement

LB-IPAM

Cilium LB-IPAM provides IP address management for Kubernetes LoadBalancer Services.

In this lab, the available addresses come from a dedicated pool:

The allocation process looks like this:

LB-IPAM can assign:

Service → 192.168.122.241

but that does not automatically make the address reachable from the LAN.

The network still needs to know where 192.168.122.241 exists.

L2 Announcement

The LoadBalancer IP belongs to the same Layer 2 network as the Kubernetes nodes.

When an external client wants to communicate with 192.168.122.241, it needs to resolve that IP to a MAC address using the local network's address-resolution mechanism.

Cilium's L2 Announcement functionality allows a Kubernetes node to announce that it can receive traffic for the LoadBalancer IP.

Expressing the L2 Policy

The L2 policy in this lab specifies which LoadBalancer addresses should be announced and which node interfaces may be used for that announcement.

The relevant configuration is:

loadBalancerIPs: true

interfaces:
  - ^ens3$

The policy expresses two decisions:

This allows Cilium to connect the Kubernetes LoadBalancer abstraction with the actual Layer 2 network interface through which the address becomes reachable.

Testing External Connectivity

At this point, the complete external path can be tested from outside the Service abstraction:

curl http://192.168.122.241

If the request succeeds, several components have successfully worked together:

Cilium Ingress and Envoy

Cilium can allocate a LoadBalancer IP through LB-IPAM, advertise that address on the local network using L2 Announcement, and forward traffic into the Kubernetes networking datapath.

But there is still a limitation.

A LoadBalancer address tells us where traffic enters the cluster. It does not by itself define how HTTP requests should be routed between multiple applications.

Consider a cluster running several applications:

Exposing every application independently would require a separate external entry point for each application.

Instead, we can introduce an HTTP aware entry point:

Now a single LoadBalancer address can act as the entry point for multiple applications.

The component responsible for understanding the HTTP request is Envoy.

Cilium provides an Ingress controller that translates Kubernetes Ingress configuration into the configuration required by Envoy.

Kubernetes Ingress and the Data Plane

The Kubernetes Ingress resource describes the desired HTTP routing behavior.

For example, our test application uses the hostname:

nginx.k8s.local

The Ingress resource describes the desired routing:

The Cilium Ingress Controller observes this Kubernetes configuration and configures Envoy accordingly.

Envoy then becomes the component that actually handles the HTTP request.

The important point is:

The Ingress resource describes the desired HTTP routing, while Envoy becomes the data-plane component handling the request.

This is different from the Service datapath we examined earlier.

At the Service layer, Cilium can make forwarding decisions based on network-level information such as IP addresses and ports.

At the Ingress layer, the system needs to understand application-level information such as the HTTP Host header and request path.

For example:

Host: nginx.k8s.local
Path: /

Envoy can inspect this information and determine which configured backend should receive the request.

Following One HTTP Request

Rather than looking at each component independently, we can now follow a single HTTP request through the entire system.

The request is generated with:

curl --resolve nginx.k8s.local:80:192.168.122.241 \
  http://nginx.k8s.local/

The --resolve option makes the client connect to 192.168.122.241 while still sending the HTTP request for:

Host: nginx.k8s.local

This is useful because the request now contains both pieces of information required by the architecture:

Destination IP
192.168.122.241

Host
nginx.k8s.local

We can trace the request conceptually as:

The Client Targets the LoadBalancer IP

The client starts with an HTTP request for:

http://nginx.k8s.local/

The --resolve option maps the hostname to the LoadBalancer address:

nginx.k8s.local: 192.168.122.241

The client therefore sends the packet toward the address allocated by Cilium LB-IPAM.

Local Network Resolves the Destination

Because 192.168.122.241 belongs to the local Layer 2 network, the client needs to determine which MAC address should receive the traffic.

The L2 Announcement mechanism allows Cilium to advertise the LoadBalancer address through an eligible Kubernetes node interface.

The important distinction from the previous section is now visible:

LB-IPAM provided the address. L2 Announcement made the address reachable on the network.

The Packet Enters the Kubernetes Datapath

Once the packet reaches the Kubernetes node, it enters the networking datapath managed by Cilium.

At this point, the request has crossed from the physical or virtual network into Kubernetes. The packet can now be processed according to the networking configuration established by Cilium.

Envoy Handles the HTTP Request

The request reaches the Envoy based Ingress path.

Unlike a simple L3/L4 forwarding decision, Envoy can inspect HTTP information such as:

Host: nginx.k8s.local
Path: /

The Ingress configuration tells Cilium which HTTP routes should exist, while Envoy implements those routes in the data plane.

Envoy therefore determines that the request should be forwarded to the Service associated with the Ingress rule.

The Service Resolves the Backend

The request is then forwarded toward the Kubernetes Service:

The Service provides the stable abstraction between the Ingress layer and the actual application Pods.

The client does not need to know which Pod is running the application.

Cilium Selects the Backend

With kube-proxy removed, Cilium's eBPF-based Service handling provides the forwarding mechanism for the Service.

The request can therefore reach one of the NGINX Pods behind the Service.

The Complete Path

This single request connects the different networking mechanisms explored throughout the article.

1. The request starts as ordinary HTTP traffic on the network.

2. Cilium L2 Announcement makes the LoadBalancer address reachable.

3. Cilium datapath brings the traffic into the cluster.

4. Envoy interprets the HTTP request and applies the Ingress routing configuration.

5. The request then reaches a Kubernetes Service, where Cilium's eBPF Service datapath selects the backend workload.

Conclusion

Kubernetes networking becomes easier to understand when we stop looking at Pods, Services, LoadBalancers, and Ingress as isolated Kubernetes objects and instead follow the packet moving between them.

In this lab, we started with basic Pod networking and Service connectivity, then progressively changed the networking datapath by introducing Cilium's kube-proxy replacement, LB-IPAM, L2 Announcement, and Envoy-based Ingress.

Each component solves a different part of the networking problem:

Together, these components form a layered networking architecture:

This also gives us a more useful mental model of Cilium.

Cilium is not simply the component that gives Pods IP addresses. It connects Kubernetes state with the Linux networking datapath and provides mechanisms for processing traffic at different layers. Some decisions can be handled in the kernel through eBPF, while application-layer traffic can be handled by Envoy in userspace.

Ultimately, the most useful way to understand Kubernetes networking is to ask a simple question whenever traffic needs to move:

What component is responsible for making the next networking decision?

Once that question can be answered, Kubernetes networking becomes less about memorizing individual components and more about understanding the datapath that connects them.