Part-1 How to setup a Kubernetes Cluster on AWS EC2 using Kubeadm, containerd and calico

Gnanaprakash Ramswami — Sat, 12 Mar 2022 17:42:21 +0000

In this article we will see how to setup a k8s cluster on AWS EC2 hosts using kubeadm.

We will see in a series of steps

Where to start?
What applications to install?
How do we install these applications?

Where and what applications?

Let us consider a basic cluster setup with 1 master node and 2 worker nodes.

The master node runs the Control Plane and the worker nodes run the actual workloads

Install these 2 apps on both master and worker nodes

Container runtime
Kubelet

These 2 apps run as regular linux processes. They can be installed from the package repository like any other linux packages

They need to be installed on all the nodes.

Master Node

The master node runs the Control Plane along with the other necessary apps

Control plane consists of the following apps which run as Pods.

API server
Controller Manager
Scheduler
Etcd

Along with the above we need to install Kubeproxy which also runs as a Pod.

Worker Node

Each of the worker nodes also should run Kubeproxy.

Now, what are the main things required to deploy these Pods.

Kubernetes manifest files for each of the apps
Security setup to prevent unauthorized access to the apps and securing the communication between the apps

How do we setup the apps in the master node? Because we said all the control plane apps run as Pods. Without the control plane in place how are we going to deploy and run these Pods?

To solve this problem, we have something called Static Pods.

Static Pods

These are pods managed directly by the kubelet daemon. So, the request for deploying a pod does not go through the API server in the control plane which is the case for any regular pod.

How does kubelet daemon do this?

Kubelet looks for any manifests available in a specific location on the Node where it is running.

/etc/kubernetes/manifests
If the kubelet finds a Pod manifest in that location, it schedules a Pod directly on the node without having to go through the API server on the control plane running on the master node.

Kubelet also restarts the static pods in case if the pods crash.

Creating static Pod manifests

All static pod manifests have to be created and dropped into this location /etc/kubernetes/manifests on their respective nodes.

Securing the pods

Securing the communication between the apps within the cluster is one of the key aspects.

K8s uses TLS certificate based security for this purpose.

In order to decide which certificate will be used in each case we need to understand the communication flow between various components.

First, most of all the components talk to the API server as it is the centre of all the communications. So, every other app that wants to talk to API server needs to send a TLS certificate.

For learning purposes we can generate self-signed CA certificates for all the different apps.

As API server is the one which receives requests, we need to setup server certificate. For all other apps which connect to API server, client certificates are to be setup.

In cases where the API server connects to other apps, client certificate is required for API server too.

Server and client certificate for API server
Client certificate for Controller manager and Scheduler
Server and client certificate for Kubelet
Server certificate for Etcd

The certificates have to be stored on the respective nodes in this location /etc/kubernetes/pki

We also need to setup a certificate for the K8s admin users who will access the K8s cluster through API or CLI. This will be a client certificate.

We need to generate a self-signed certificate which will be the root. Then generate certs for all the apps and sign them with the root.

So, in order to get the k8s cluster up and running, we need to setup all these stuff.

All mandatory process for master and worker nodes
Manifest files for all pods
Certificates for securing the apps

We know by experience that these can take a lot of time if we do it manually. Luckily there is a CLI tool called Kubeadm.

Kubeadm

Kubeadm is a tool created and maintained by Kubernetes. A very detailed documentation is available for this tool.

This CLI tool helps in bootstrapping the k8s cluster by creating the necessary artifacts and performing the necessary actions.

Prerequisites

For this session, we are going to setup our k8s cluster on AWS infrastructure. It is fine if you have a preference to use your own hardware/VM or any other cloud hosted infra.

You can refer to kubeadm document here

The following are the infrastructure requirements

Any Linux based hosts. We will be using the latest Ubuntu VM available on AWS EC2.
Capacity of the servers - 2 CPUs and 2 GB or greater. We will choose t2.medium type as it provides 2 vCPU and 4 GiB memory.We will be creating 1 master node and 2 worker nodes and all 3 nodes will use the t2.medium type instances.
All instances should be part of the same network. In this case all the EC2 instances will be part of a single VPC.
Should have unique host name and MAC address. All EC2 instances will have unique hostnames.
Required ports have to be opened. Please use the details here to configure open the required ports.
Memory swapping should be disabled. Kubelets are designed to run the pods within the memory available on the host. Keeping swap enabled may destabilise the k8s cluster.

Now the we understand the pre-requisites, we can start the installation!

How do we install?

Step-1 : Setup the hosts

We need 3 hosts for creating the k8s cluster. We will be using AWS EC2 instances. You can sign up for AWS Free tier. Please read through the free tier details. Once you have setup the k8s cluster and finished practicing, be sure to delete the EC2 instances so that you don't incur any unexpected cost.

Creating hosts

Sign up for a new AWS Free tier account if you don't have one.
Create 3 t2.medium EC2 instances
Name 1 instance as master and the other 2 instances as worker1 and worker2.
Launch the instances. Don't forget to download the key-pair pem file as you will need this to ssh into the instances . Refer AWS documentation here

Note:- You can use the same keypair to access all the instances.

Once you launch a Public IP will be assigned to each of those instances. Make sure you are able to SSH to the instances using the public IP. Use the following command from your laptop terminal to login.

ssh -i k8s-cluster.pem ubuntu@54.179.167.247

Setting up pre-requisites

Disable memory swapping

SSH to each of the instances and execute this command.

sudo swapoff -a

Open required ports

By default AWS opens only port 22 for incoming SSH connections. So in order for all the k8s components running on 3 instances to communicate to each other, other ports need to be opened as per this kubeadm reference

Use the AWS security groups to configure the inbound ports. Refer to AWS EC2 Security Groups

So far, we have provisioned the infrastructure and configured the necessary pre-requisites.

Step-2 : Setup container runtime

Container runtime has to be installed on master node and all the worker nodes.

We will be using containerd as the container runtime as this is the standard recognised and used by all the managed k8s cluster providers EKS, AKS and GKE

Please refer to Container Runtime Interface(CRI) if you want to refresh the concepts around the CRI.

Ok. Let us install the container runtime.

We will be following the steps given here.

The container installation steps are captured in a script file that can be found here

Run the install script on all the nodes.

Step-3 : Install kubeadm, kubelet and kubectl

Now we need to install kubeadm, kubelet and kubectl on all the nodes.

As mentioned earlier kubeadm will be used to initialise the cluster. It will create the necessary components for the control plane.

Kubelet is required for starting pods and containers

Kubectl is required for communicating with the cluster

We will be following the steps given here.

The installation steps are captured in a script file that can be found here

Run the above script on all the nodes.

After completing the installations, check the status of each of the components by executing these commands

kubeadm version
kubectl version
service kubelet status

Step-3 : Initialise the cluster

We will be using the kubeadm command to initialise the cluster.

On the master node, execute the following command.

sudo kubeadm init

Once the initialisation completes successfully, you should see some output like the following.

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.31.17.66:6443 --token i1yboi.66y4mulp2gtblrvv \
    --discovery-token-ca-cert-hash sha256:4ddfa86a2cf2df7aaf21858c685e98ee2d9687252e2a2ef46df0c96223671504

In the output you will notice the different steps that were executed to setup the control plane and other processes.

First executes preflight step to check and pull the necessary images.

[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'

Creates the certificates for all the k8s components.

[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master] and IPs [10.96.0.1 172.31.17.66]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master] and IPs [172.31.17.66 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master] and IPs [172.31.17.66 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key

Creates the necessary kube config files that connect to the API server.

[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file

Configures the kubelet config and starts the kubelet service

[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet

Generates the manifest files for static pods and starts the pods.

[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"

Sets up the manifest for etcd

[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"

Creates a ConfigMap called kubeadm-config in the kube-system namespace. This configuration is later used during kubeadm join, kubeadm-reset and kubeadm-upgrade

[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace

A kubelet config is created in kube-system namespace for all the kubelets in the cluster

[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster

Adds a taint to prevent any workload pods from being scheduled on the master node.

[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

Then, creates bootstrap tokens to be used along with RBAC roles to enable authentication in the API server and used when a new cluster is created or joining new nodes to the cluster.

[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace

Sets up CoreDNS as the DNS server and kube-proxy which is a network proxy that contains network rules on nodes and allows the traffic to the pods from inside and outside of cluster.

[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

With this, we have completed the cluster initialisation step!!!

Step-4 : Connecting to the cluster

We know that kubectl is the CLI tool to talk to the cluster. So, how does kubectl know to connect to the cluster.

Let us do a quick check. Execute this command on the master node.

ubuntu@master:~$ kubectl get nodes
The connection to the server localhost:8080 was refused - did you specify the right host or port?

kubectl is unable to connect as the kubeconfig is not setup yet.

To make it work, we need to copy the kubeconfig to a default location on the node from where kubectl can use it.

Execute the following commands.

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Now, if you check the nodes, you should see an output like this.

ubuntu@master:~$ kubectl get nodes
NAME     STATUS     ROLES                  AGE   VERSION
master   NotReady   control-plane,master   11h   v1.23.4

The status of the master node is showing as 'NotReady' because the pod networking is not setup. So let us setup the pod network in the next step.

Step-5 : Setup pod network

You can refer to k8s networking to understand the fundamental concepts.

There are a number of networking solutions available. For this session we are going to use calico.

Let us follow these steps to install calico on k8s. This installation has to be done on the master node.

Pre-requisites

Check the calico network requirements

In this session, we are using the default BGP bidirectional networking. For all the nodes, configure the inbound rules in security groups to allow TCP port 179. Only allow the VPC IP range for eg 172.31.0.0/16.

Installation

Download the calico manifest file.

curl https://projectcalico.docs.tigera.io/manifests/calico.yaml -O

By default calico is configured to use 192.168.0.0/16 for the pod network. This should not overlap with the node IP range. If you are using this IP range already, make the necessary changes to the calico manifest.

Uncomment these 2 lines in calico.yaml file if you want to use the default CIDR range.

- name: CALICO_IPV4POOL_CIDR value: "192.168.0.0/16"

Now, apply the manifest.

kubectl apply -f calico.yaml

After the installation completes, you can verify by checking the mode status and the calico pods created and running.

kubectl get nodes
NAME     STATUS   ROLES                  AGE   VERSION
master   Ready    control-plane,master   3d    v1.23.4

Checking the pods.

kubectl get pod -n kube-system
calico-kube-controllers-566dc76669-dcnbq   1/1     Running   0          56s
calico-node-nkhj9                          1/1     Running   0          56s
coredns-64897985d-28rdb                    1/1     Running   0          3d
coredns-64897985d-82m9d                    1/1     Running   0          3d
etcd-master                                1/1     Running   0          3d
kube-apiserver-master                      1/1     Running   0          3d
kube-controller-manager-master             1/1     Running   0          3d
kube-proxy-d25qg                           1/1     Running   0          3d
kube-scheduler-master                      1/1     Running   0          3d

Notice the calico pods running and coredns pods in running state.

We have successfully created a pod network for a single node cluster. We still need to bring the worker nodes into the pod network. To do this we need to join the worker nodes to the cluster.

Step-6 : Add worker nodes to cluster

Use the kubadm join command shown in the result of Step-3 to join the worker nodes to cluster.

kubeadm join 172.31.17.66:6443 --token l87fxe.30820pct4o7k80ax --discovery-token-ca-cert-hash sha256:4ddfa86a2cf2df7aaf21858c685e98ee2d9687252e2a2ef46df0c96223671504

After execution of the command, you can verify if the nodes are joined in the cluster by executing the following in master node.

ubuntu@master:~$ kubectl get nodes
NAME      STATUS     ROLES                  AGE     VERSION
master    Ready      control-plane,master   3d1h    v1.23.4
worker1   Ready      <none>                 3m53s   v1.23.4
worker2   NotReady   <none>                 18s     v1.23.4

Step-7 : Checking calico networking is setup for all nodes

You can see 3 calico-node pods are running.

ubuntu@master:~$ kubectl get pod -A  -o wide
NAMESPACE         NAME                                       READY   STATUS    RESTARTS   AGE    IP               NODE      NOMINATED NODE   READINESS GATES
kube-system       calico-kube-controllers-566dc76669-fss4z   1/1     Running   0          30s    172.16.189.65    worker2   <none>           <none>
kube-system       calico-node-b7mwj                          1/1     Running   0          31s    172.31.28.225    worker2   <none>           <none>
kube-system       calico-node-ln7g2                          1/1     Running   0          31s    172.31.17.66     master    <none>           <none>
kube-system       calico-node-qkg8c                          1/1     Running   0          31s    172.31.22.174    worker1   <none>           <none>
kube-system       coredns-64897985d-28rdb                    1/1     Running   0          4d9h   192.168.219.65   master    <none>           <none>
kube-system       coredns-64897985d-82m9d                    1/1     Running   0          4d9h   192.168.219.66   master    <none>           <none>
kube-system       etcd-master                                1/1     Running   0          4d9h   172.31.17.66     master    <none>           <none>
kube-system       kube-apiserver-master                      1/1     Running   0          4d9h   172.31.17.66     master    <none>           <none>
kube-system       kube-controller-manager-master             1/1     Running   0          4d9h   172.31.17.66     master    <none>           <none>
kube-system       kube-proxy-d25qg                           1/1     Running   0          4d9h   172.31.17.66     master    <none>           <none>
kube-system       kube-proxy-nqwkm                           1/1     Running   0          32h    172.31.28.225    worker2   <none>           <none>
kube-system       kube-proxy-nvn4d                           1/1     Running   0          32h    172.31.22.174    worker1   <none>           <none>
kube-system       kube-scheduler-master                      1/1     Running   0          4d9h   172.31.17.66     master    <none>           <none>

Let us create a nginx pod to check if the IP allocation works as expected.

Create a nginx Pod manfiest.

apiVersion: v1
kind: Pod
metadata:
   name: nginx
   namespace: default
spec:
   containers:
   - name: nginx
     image: nginx
     ports:
       - containerPort: 80

Create the nginx pod

kubectl create -f nginx.yaml

View the pod details

kubectl get pods -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP               NODE      NOMINATED NODE   READINESS GATES
nginx   1/1     Running   0          13s   192.168.189.65   worker2   <none>           <none>

You can see the newly created nginx pod is assigned an IP from the CIDR block 192.168.0.0/16

Let us create another pod using another simpler method.

kubectl run nginx2 --image=nginx

View the pod details now

kubectl get pods -o wide
NAME     READY   STATUS    RESTARTS   AGE   IP                NODE      NOMINATED NODE   READINESS GATES
nginx    1/1     Running   0          57m   192.168.189.65    worker2   <none>           <none>
nginx2   1/1     Running   0          7s    192.168.235.130   worker1   <none>           <none>

You can see the new pod 'nginx2' is scheduled on worker1 and the IP is within the CIDR range 192.168.0.0/16

With this we have completed a basic k8s cluster setup and tested it.

In the next article we will deploy a sample "go lang" application.

DEV Community: Gnanaprakash Ramswami