microk8s up and running, the next step is to add new nodes to the Kubernetes cluster. Adding nodes not only increases the cluster's computing resources, but also adds redundancy. Redundancy, in turn, increases the availability and reliability of the cluster because the Kubernetes Pods, e.g., the replicas of the same service, can be distributed over multiple hardware devices. Multiple hardware instances mean that failures can be tolerated. That's exactly the reason why the Kubernetes scheduler tries to ensure that Pods from the same application are distributed onto different machines.
This post is a step-by-step guide for adding new nodes to a
microk8s Kubernetes cluster running on Raspberry Pi (RPi) microcomputers. In particular, we will discuss how to add RPi nodes running the Ubuntu 20.04 release. As you will see, Ubuntu 20.04 nodes require a little manual tweaking related to
cgroups before you can actually use them in an
microk8s Kubernetes cluster.
Note: In the remainder of this post, all shell commands are executed on the RPi minicomputers. For me, the most convenient way to manage these microcomputers is to
ssh into them from my development laptop. In case you'd like to know how to enable
ssh, the official Raspberry Pi documentation on remote access via
ssh is a good place to start.
Luckily for us, Kubernetes makes adding new nodes really easy. In case of
microk8s, the shell command
microk8s add-node generates a connection string and outputs a list of suggested
microk8s join commands for adding new nodes to the current
microk8s cluster. The cluster node on which this command is executed becomes the main node of the Kubernetes cluster and will host its control plane.
So, to get started, connect to the main Raspberry Pi|s (i.e., the RPi you have designated to become the main node of your Kubernetes cluster) and issue:
ubuntu@ubuntu:~$ microk8s add-node Join node with: microk8s join 192.168.178.66:25000/ipySurdsIOGAlYyYQzzOOSlpUTEDNGfk If the node you are adding is not reachable through the default interface you can use one of the following: microk8s join 192.168.178.66:25000/ipySurdsIOGAlYyYQzzOOSlpUTEDNGfk microk8s join 10.1.79.0:25000/ipySurdsIOGAlYyYQzzOOSlpUTEDNGfk microk8s join 172.17.0.1:25000/ipySurdsIOGAlYyYQzzOOSlpUTEDNGfk ubuntu@ubuntu:~$
In the above output of the
microk8s add-node command you see the string
ipySurdsIOGAlYyYQzzOOSlpUTEDNGfk that is required for a new node to join the Kubernetes cluster. The output also gives multiple suggestions for the parameters of the
microk8s join command, which is great if you should have any issues with your setup. (Note: the value of your token will be different since Kubernetes generates it individually for every cluster).
Connect to the RPi minicomputer that you want to add to the
microk8s cluster. The
microk8s software must be installed and configured on this RPi before you proceed. If that's not the case, check the "Installing microk8s and nginx on Raspberry Pi 4" post for a step-by-step installation guide.
If you installed the operating system on your Raspberry Pi from a standard Linux image (in my setup, I use the Ubuntu 20.04 image for my Raspberry Pi), you might want to change the hostname of your second Raspberry Pi device to something unique (otherwise, you will see multiple Kubernetes nodes all having the same name
ubuntu later on in your Kubernetes dashboard/command line). Under Linux, you can change the hostname like this:
/etc/hostnameand replace the current hostname with the new hostname
/etc/hostsand replace every occurrence of the current hostname with the new hostname
- reboot the system by issuing
In my case, the hostname of the second RPi is
ubuntu-2. To add the second RPi to the
microk8s cluster, I need to
ubuntu-2 and issue the command:
ubuntu@ubuntu-2:~$ microk8s join 192.168.178.66:25000/ipySurdsIOGAlYyYQzzOOSlpUTEDNGfk ubuntu@ubuntu-2:~$
You can verify that the new node has been successfully added to your Kubernetes cluster by logging into the Raspberry Pi running the main cluster node (in my setup, the RPi with the hostname
ubuntu) and issuing:
ubuntu@ubuntu:~$ microk8s.kubectl get nodes NAME STATUS ROLES AGE VERSION ubuntu Ready <none> 59d v1.18.3 ubuntu-2 NotReady <none> 98s v1.18.3 ubuntu@ubuntu:~$
As you can see from the output above, we now have two nodes -
ubuntu-2 - in our
microk8s Kubernetes cluster. However, we have an issue that needs to be fix: The newly added node is in the state
NotReady. It turns out, this is not due to a timing issue - even after more than an hour the node remains in this state.
After searching a bit, I stumbled upon the Konstantinos Tsakalozos comment on this issue in the
microk8s GitHub project.
K8s was complaining that the cgroup memory was not enabled. Looking at /proc/cgroups was showing memeory disabled. Changes in config.txt are not enough, what actually worked is editing /boot/firmware/nobtcmd.txt, appending cgroup_enable=memory cgroup_memory=1. and rebooting.
/proc/cgroups file on my second RPi (the one I want to add to the
microk8s cluster) confirmed this:
ubuntu@ubuntu-2:~$ sudo cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset 8 1 1 cpu 7 1 1 cpuacct 7 1 1 blkio 3 1 1 memory 0 56 0 devices 5 47 1 freezer 6 2 1 net_cls 10 1 1 perf_event 9 1 1 net_prio 10 1 1 pids 2 52 1 rdma 4 1 1 ubuntu@ubuntu-2:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04 LTS Release: 20.04 Codename: focal ubuntu@ubuntu-2:~$ uname -a Linux ubuntu-2 5.4.0-1012-raspi #12-Ubuntu SMP Wed May 27 04:08:35 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
cgroups man page explains that Linux control groups, usually referred to as
cgroups, are a Linux kernel feature which allow processes to be organized into hierarchical groups whose resource usage can then be limited and monitored. In a nutshell, as Tim Downey puts it in his post,
cgroups allow you to do things like limit the amount of CPU usage of a process, limit the amount of memory that a process can consume, control how many additional processes a process can fork, and even “freeze” a process in place. In other words,
cgroups are a feature provided by the Linux kernel to manage, restrict, and audit groups of processes. Compared to other approaches like the
nice command or
cgroups are more flexible as they can operate on (sub)sets of processes (possibly with different system users).
cgroups provide a mechanism for aggregating/partitioning sets of tasks, and all their future children, into hierarchical groups with specialized behavior. A container orchestrator like Kubernetes uses
cgroups to place resource limits on the containers it creates. In case you want to learn more about
cgroups, check out this documentation on kernel.org.
Coming back to our issue, in contrast to the RPi running Ubuntu 20.04, the
cgroups settings on my main RPi (the main Kubernetes node) look like this:
ubuntu@ubuntu:~$ sudo cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset 9 33 1 cpu 4 111 1 cpuacct 4 111 1 blkio 2 111 1 memory 3 125 1 devices 7 111 1 freezer 11 34 1 net_cls 6 33 1 perf_event 8 33 1 net_prio 6 33 1 pids 5 117 1 rdma 10 1 1 ubuntu@ubuntu:~$ uname -a Linux ubuntu 5.3.0-1017-raspi2 #19~18.04.1-Ubuntu SMP Fri Jan 17 11:14:07 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux ubuntu@ubuntu:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.4 LTS Release: 18.04 Codename: bionic ubuntu@ubuntu:~$
Thus, as described in this post, you have to edit the
/boot/firmware/cmdline.txt file to look like this:
net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 console=tty1 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc
by adding the
cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1
Finally, you need to reboot the RPi for the changes to take effect. Probably the simplest way to do this is by issuing the following command :
ubuntu@ubuntu-2:~$ sudo reboot
After the reboot is complete and you have logged in into the second RPi, you can check the
cgroups settings and should see the
memory cgroup enabled:
ubuntu@ubuntu-2:~$ cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset 11 6 1 cpu 4 6 1 cpuacct 4 6 1 blkio 3 6 1 memory 9 101 1 devices 5 52 1 freezer 6 7 1 net_cls 7 6 1 perf_event 2 6 1 net_prio 7 6 1 pids 8 57 1 rdma 10 1 1 ubuntu@ubuntu-2:~$
As a result, if you now
ssh into the Raspberry Pi running the main
microk8s cluster node and check the node status in the cluster, you should see something like this:
ubuntu@ubuntu:~$ microk8s.kubectl get nodes NAME STATUS ROLES AGE VERSION ubuntu Ready <none> 59d v1.18.3 ubuntu-2 Ready <none> 4h12m v1.18.3 ubuntu@ubuntu:~$
Voilà - as you can see, our
microk8s Kubernetes cluster is now composed of two nodes, i.e., two RPi minicomputers. Both Kubernetes nodes are now in the
Ready state. You can verify this in the Kubernetes dashboard (in case you haven't setup the dashboard yet, take a look at this post):
To recap, the three steps needed to add a RPi running Ubuntu 20.04 to your
microk8s Kubernetes cluster look like this:
- connect to the RPi you want to add to your
microk8sKubernetes cluster and edit the
/boot/firmware/cmdline.txtfile by adding
cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1to enable
cgroup memory. Restart that RPi (e.g., by issuing
- generate the connection token on your main
microk8snode using the
- add the second RPi to the
microk8scluster using the
microk8s joincommand with the token obtained in step 2