Building a Kubernetes cluster out of 3 Raspberry Pis

Based off of the great the post https://kubecloud.io/setting-up-a-kubernetes-1-11-raspberry-pi-cluster-using-kubeadm-952bbda329c8
and the YouTube video https://www.youtube.com/watch?v=2YoK4bBy3CM, for networking.

Hardware

For the hardware

1. 3 RaspberryPi 4B, with 4GB RAM
2. Dorhea Raspberry Pi 4 B Case
3. A set of aluminum heatsinks
4. TP-LINK 8-port Gigabit Desktop Switch
5. A set of Cat6 Ethernet Cables
6. Set of USB C charging cables
7. Anker 60W 6-Port USB wall charger
8. Set of Samsung 32GB micro SDHC cards

Setting up the Raspberry Pis

In this setup I am using 3 Raspberry Pi 4s, but other versions of the Pis should work as well. To set them up:

1. Flash 3 SD Cards with the latest Raspian images.
2. After flashing, remount the SD Card and in the /boot directory of the SD Card add an empty file named ssh. This will enable ssh on the Pi running off of this image.
3. Unmount the SD Card and insert it into a Pi.

Network and Kubeadm Installs

Connect the Pis to a switch and the switch to your local network. When the Pi boots, it can be reached at ssh [e-mail] with the password raspberry. Now since all three Pis start with the same DNS name, I would power on one, updated its network settings, and rebooted it. After it is up and running with it's own DNS name, I could then plugin the next Pi and repeated the process. Stepping through Pis this way keeps only one Pi with the domain name raspberrypi.local on the network at a time.

Network Setup

We start by updating the Pi's /etc/hosts and /etc/dhcpcd.conf. To do this, I used the script below, passing in the hostname and last octet of the IP address as the parameters.

#!/bin/sh
#
# set_host.sh
#
# Usage: ./set_host.sh <hostname> <last octet>
#  ./set_host.sh k8master 223
# for example
#

hostname=$1
ip=$2
dns=192.168.1.1

sudo -- sh -c "cat <<EOT >> /etc/hosts
192.168.1.223   k8master
192.168.1.224   k8worker1
192.168.1.225   k8worker2
EOT"


sudo cat <<EOT >> /etc/dhcpcd.conf
interface eth0
static ip_address=192.168.1.${ip}/24
static routers=${dns}
static domain_name_servers=${dns}
EOT

sudo sed -i s/raspberrypi/${hostname}/g /etc/hosts
sudo hostnamectl --transient set-hostname ${hostname}
sudo hostnamectl --static set-hostname ${hostname}
sudo hostnamectl --pretty set-hostname ${hostname}

After running the script, be sure to reboot the Pi, for the network changes to take effect.

Software Install

After the network on the Pi setup, installation of docker and Kubernetes software is next. Again a shell script was used for installation.

#!/bin/sh
#
# Usage: ./set_kube.sh
# InstalLs the docker and kubernetes software
#

# Install Docker
curl -sSL get.docker.com | sh && \
sudo usermod pi -aG docker

# Disable Swap
sudo dphys-swapfile swapoff && \
sudo dphys-swapfile uninstall && \
sudo update-rc.d dphys-swapfile remove
echo Adding " cgroup_enable=cpuset cgroup_enable=memory" to /boot/cmdline.txt
sudo cp /boot/cmdline.txt /boot/cmdline_backup.txt
orig="$(head -n1 /boot/cmdline.txt) cgroup_enable=cpuset cgroup_enable=memory"
echo $orig | sudo tee /boot/cmdline.txt

# Add repo list and install kubeadm
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && \
sudo apt-get update -q && \
sudo apt-get install -qy kubeadm

Start up

Only on your master node, run kubeadm init, and include the pod-network-cdir parameter, needed for the flannel networking. The pod-network-cidr needs to be different then the one for your node's network.

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

As kubeadm runs, it will print out needed information.

Setup for kubectl:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Connection command for worker nodes

  kubeadm join 192.168.1.223:6443 --token k3mpju.xxxxxxxloccv878b \
    --discovery-token-ca-cert-hash sha256:63225cc63b7189c3cc4c091fff07cad4bxxxxxxxxxx2846f36cd00543ad6f54c

Before adding worker nodes install the Flannel network.

Install Flannel network

On the master node, run the command below to install networking.

kubectl apply -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml

Adding Worker Nodes

Log into each worker node, and using the connection command displayed when the master's kubeadm was executed, add the workers to the cluster.

sudo kubeadm join 192.168.1.223:6443 --token k3mpju.xxxxxxxloccv878b \
    --discovery-token-ca-cert-hash sha256:63225cc63b7189c3cc4c091fff07cad4bxxxxxxxxxx2846f36cd00543ad6f54c

If you lose the join token, you can recreate it with these commands.

kubeadm token generate
kubeadm token create <generated token> --print-join-command --ttl=0

Conclusion

At the end of this process, you should have three nodes Ready along with set of pods ready as well.