DEV Community

A-LPHARM
A-LPHARM

Posted on

Exploring Cilium Network Integration with AWS EKS

Today, I explore how the Cilium network works by integrating it into AWS EKS, which has been quite intriguing. Creating and managing clusters with Cilium improves network connectivity, acting as a network superhero. We can leverage the add-on modules provided by EKS.
In this blog post, we will explore and test how to integrate the Cilium networking add-on directly with EKS. In the next blog, we will dive further into the newer version of AWS EKS cluster creation flexibility, which promises to simplify this integration.
Every EKS cluster comes with default networking add-ons, including AWS VPC CNI, CoreDNS, and kube-proxy, to enable pod and service operations in the EKS clusters. In our cluster deployment, we will follow the documentation where the taints provided control the scheduling of application pods to nodes based on the readiness status of Cilium. As of this writing, this is limited to IPv4.
Here is an example of a cluster configuration:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: henry-eks
  region: us-east-1

managedNodeGroups:
- name: ng-1
  desiredCapacity: 2
  privateNetworking: true
  # taint nodes so that application pods are
  # not scheduled/executed until Cilium is deployed.
  # Alternatively, see the note below.
  taints:
   - key: "node.cilium.io/agent-not-ready"
     value: "true"
     effect: "NoExecute"
Enter fullscreen mode Exit fullscreen mode

Once the cluster is created the networking add-on AWS VPC CNI plugin is responsible for setting up the virtual network devices as well as for IP address management via ENI. once the cilium CNI plugin is set up it attaches the eBPF programs to the network devices set up by the AWS VPC CNI plugin in other to enforce network policies, perform load-balancing and encryption.

To confirm the AWS VPC CNI the version 1.16.0 you are using to guarantee compatible with cilium

kubectl -n kube-system get ds/aws-node -o json | jq -r '.spec.template.spec.containers[0].image'
602401143452.dkr.ecr.us-east-1.amazonaws.com/amazon-k8s-cni:v1.16.0-eksbuild.1

Before we go deeper into installing Cilium in the EKS cluster, I'll discuss another feature, AWS ENI (Elastic Network Interface). It is an allocator and a virtual network interface that we can attach to any nodes on our cluster, necessary for allocating IP addresses needed by communicating with the EC2 instance API. Once Cilium is set up, each node creates a Cilium CRD matching the node name ciliumnodes.cilium.io, which also creates the ENI parameters by communicating with the EC2 metadata API to retrieve the instance ID and VPC information.

now cilium will manage the AWS ENI instead of the VPC CNI, so the aws-node daemonset must be patched to prevent conflicting behavior:

kubectl -n kube-system patch daemonset aws-node --type='strategic' -p='{"spec":{"template":{"spec":{"nodeSelector":{"io.cilium/aws-node-enabled":"true"}}}}}'
Enter fullscreen mode Exit fullscreen mode

once it has been patched

then we work on installing cillium on the Eks cluster, using helm

helm repo add cilium https://helm.cilium.io/

Enter fullscreen mode Exit fullscreen mode

 then after updating and installing the cilium repo on the local machine, we then use helm install repo

 

helm install cilium cilium/cilium --version 1.15.6 \
  --namespace kube-system \
  --set eni.enabled=true \
  --set ipam.mode=eni \
  --set egressMasqueradeInterfaces=eth0 \
  --set routingMode=native

Enter fullscreen mode Exit fullscreen mode

If you created your cluster and did not taint the nodes with node.cilium.io/agent-not-ready, the unmanaged pods need to be restarted manually to ensure Cilium starts managing them. To do this:

kubectl get pods --all-namespaces -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,HOSTNETWORK:.spec.hostNetwork --no-headers=true | grep '<none>' | awk '{print "-n "$1" "$2}' | xargs -L 1 -r kubectl delete pod

Enter fullscreen mode Exit fullscreen mode

validate the installation 

here, I tried something different instead of using the documentation, I deploy a Star Wars microservices application, which has three applications: Deathstar, tiefighter, xwing. the Deathstar runs on port 80 as the cluster IP, the Deathstar service provides the landing services to the application as a whole or spaceship. the tie fighter pod and xwing pod represent similar client service, moving on. to run this,
Enter fullscreen mode Exit fullscreen mode
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/1.15.6/examples/minikube/http-sw-app.yaml

Enter fullscreen mode Exit fullscreen mode

cilium will represent every pod as an endpoint in the cilium agent. we can get the list of end-point by running this command

 

# the first node  
kubectl -n kube-system exec cilium-cxvdh -- cilium-dbg endpoint list
# the second node 
kubectl -n kube-system exec cilium-lsxht -- cilium-dbg endpoint list

Enter fullscreen mode Exit fullscreen mode

this will list all the end-point for each node

kubectl -n kube-system get pods -l k8s-app=cilium
NAME           READY   STATUS    RESTARTS   AGE
cilium-cxvdh   1/1     Running   0          119m
cilium-lsxht   1/1     Running   0          119m

$ kubectl -n kube-system exec cilium-lsxht -- cilium-dbg endpoint list
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                              IPv6   IPv4             STATUS
           ENFORCEMENT        ENFORCEMENT
143        Disabled           Disabled          54154      k8s:app.kubernetes.io/name=tiefighter                                           192.168.92.83    ready 

                                                           k8s:class=tiefighter

                                                           k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default

                                                           k8s:io.cilium.k8s.policy.cluster=default

                                                           k8s:io.cilium.k8s.policy.serviceaccount=default

                                                           k8s:io.kubernetes.pod.namespace=default

                                                           k8s:org=empire

147        Disabled           Disabled          4849       k8s:app.kubernetes.io/name=xwing                                                192.168.85.248   ready 

                                                           k8s:class=xwing

                                                           k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default

                                                           k8s:io.cilium.k8s.policy.cluster=default

                                                           k8s:io.cilium.k8s.policy.serviceaccount=default

                                                           k8s:io.kubernetes.pod.namespace=default

                                                           k8s:org=alliance

735        Disabled           Disabled          4          reserved:health                                                                 192.168.75.160   ready 

1832       Disabled           Disabled          1          k8s:alpha.eksctl.io/cluster-name=henry-eks-app                                                   ready 

                                                           k8s:alpha.eksctl.io/nodegroup-name=henry-eks-1

                                                           k8s:eks.amazonaws.com/capacityType=ON_DEMAND

                                                           k8s:eks.amazonaws.com/nodegroup-image=ami-057ddb600f3bba07e

                                                           k8s:eks.amazonaws.com/nodegroup=henry-eks-1

                                                           k8s:eks.amazonaws.com/sourceLaunchTemplateId=lt-07908f6a5332b214e

                                                           k8s:eks.amazonaws.com/sourceLaunchTemplateVersion=1

                                                           k8s:node.kubernetes.io/instance-type=t2.medium

                                                           k8s:topology.kubernetes.io/region=us-east-1

                                                           k8s:topology.kubernetes.io/zone=us-east-1b

                                                           reserved:host

2579       Disabled           Disabled          6983       k8s:app.kubernetes.io/name=deathstar                                            192.168.79.49    ready 

                                                           k8s:class=deathstar

                                                           k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default

                                                           k8s:io.cilium.k8s.policy.cluster=default

                                                           k8s:io.cilium.k8s.policy.serviceaccount=default

                                                           k8s:io.kubernetes.pod.namespace=default

                                                           k8s:org=empire

Enter fullscreen mode Exit fullscreen mode

To confirm the pods can be accessed and the labels with org=empire are allowed to connect and request landing since no rules will attached to the pods xwing and tiefighter

$ kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed

$ kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed
Enter fullscreen mode Exit fullscreen mode

thank you 

till next time

Top comments (0)