Kubernetes (K8s) Private Cloud with Raspberry Pi 4s

Deploying a Kubernetes cluster on a public cloud provider is easy, but what if you want a private bare-metal deployment? This walk-through will go through the steps I took (and why we need to do them) in order to have a private Kubernetes cluster at home.

Kubernetes enables you to have the flexibility to move your workload where it is best suited. The most common reasons I hear are; latency, performance (special hardware requirements) and security (regulation or data governance). This compliments the hybrid cloud story and in my career it has become more apparent that my customers see this as well to help them resolve issues like; cost, availability and compliance. In parallel software vendors are starting to embrace containers as a standard deployment model leading to a recent increase in requests for container solutions.

As you can see in the workflow comparison below, there is greater room for error when deploying on-premises. Public clouds provide the automation and reduces the risk of error as less steps are required. But as mentioned above, private cloud provides you more options when you have unique requirements.

Workflow comparison for deploying Kubernetes on or off premises

To help people understand some of the pitfalls and challenges with on-premises Kubernetes/Container deployments, below is a guide to help you build your own.

"A cloud you can touch!"

What you will need

Compute:

• 4 x Raspberry Pi 4 - 2GB RAM [2GB+ RAM is recommended as we are deploying K8s not K3s]

Storage:

• 4 x 32GB High Speed Sand-disk Micro-SD Cards

Network:

• 4 x Raspberry Pi POE HAT [Optional if you don't want to provide USB power to the Raspberry Pi]
• 1 x Network Switch [POE Capability is only required if using Raspberry Pi POE HAT]
• 1 x Network Router
• 5 x Ethernet Cables
• 1 x Keyboard, HDMI, Mouse (for initial setup only)

Other:

• DNS Server [Optional if you want to provide round robin infrastructure resiliency]

Walk Through

Initial Raspberry Pi Configuration

• Flash Raspbian
  Refer to the Raspbian Installation guide by Raspberry Pi Foundation
  https://www.raspberrypi.org/documentation/installation/installing-images/README.md
  In short:

  • Download image from the link above
  • Use balenaEtcha to flash image onto micro-SD card

• Perform Initial Setup on Boot on startup screen (this will require keyboard, video and mouse to be connected)
  
  

Source: RaspberryPi.org

• Choose Country, Language, Timezone
• Define new password for user 'pi'
• Connect to WiFi or skip if using ethernet
• Skip update software (this caused my Raspberry Pi to hang, not sure if there's currently a bug. We will perform this activity manually later).

• Choose restart later

  • Configure Additional Settings Click the Raspberry Pi icon (top left of screen) > Preferences > Raspberry Pi Configuration

• System

  • Configure Hostname
  • Boot: To CLI

• Interfaces

  • SSH: Enable

• Choose restart later

  • Configure Static Network Perform one of the following:

• Define Static IP on Raspberry Pi: Right Click the arrow logo top right of screen and select 'Wireless & Wired Network Settings'

• Define Static IP on DHCP Server: Configure your DHCP server to define a static IP on the Raspberry Pi Mac Address.

  • Reboot and Test SSH

• Username: pi

• Password: Defined in step 2 above

• On Terminal: ssh pi@[IP Address]

• Repeat steps for all of the Raspberry Pis.

Kubernetes Cluster Preparation (via SSH)

• Perform Updates
  • apt-get update: Updates the package indexes
  • apt-get upgrade: Performs the upgrades

  sudo apt-get update
  sudo apt-get upgrade
  sudo reboot

• Configure Net.IP4.IP configuration Edit sudo vi /etc/sysctl.conf, uncomment net.ipv4.ip_forward = 1 and add net.ipv4.ip_nonlocal_bind=1.
  • Note: This is required to allow for traffic forwarding, for example Node Ports from containers to/from non-cluster devices.

sudo reboot

• Install Docker

curl -sSL get.docker.com | sh && sudo usermod pi -aG docker

• Disable Swap
  • You can verify this before/after reboot with the top command, on the top left corner next to MiB Swap should be 0.0.

sudo systemctl disable dphys-swapfile.service
sudo reboot

SSH Session Screen after `top` command

• Install Kubernetes
  • Currently forcing the previous version (1.15.3), ran into compatibility issues with the most recent version (1.16).
  • There shouldn't be any errors, however during my installation the repos were down and I had to retry in a few hours.

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | \
sudo apt-key add - && echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | \
sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt-get update -q

sudo apt-get install -qy kubelet=1.15.3-00 kubectl=1.15.3-00 kubeadm=1.15.3-00

• Repeat steps for all of the Raspberry Pis.

Kubernetes Master Node Configuration

Note: You only need to do this for the master node (in this deployment I recommend only 1 master node). Each Raspberry Pi is a node.

• Initiate Master Node

sudo kubeadm init

• Enable Connections to Port 8080
  • Without this Kubernetes services won't work

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

• Add Container Network Interface (CNI)
  • I've chosen to use Weaver, however you can get others working such as Flannel (I've verified this works with this cluster)

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

• Get Join Command
  • This will be used in the next section to join the worker nodes to the cluster. It will return something like: kubeadm join 192.168.0.101:6443 --token X.Y --discovery-token-ca-cert-hash sha256:XYZ

kubeadm token create --print-join-command

Kubernetes Worker Node Configuration

Note: You only need to do this for the worker nodes (in this deployment I recommend 3 worker node).

• Join Cluster
  • Use the join command provided at the end of the previous section

sudo kubeadm join 192.168.0.101:6443 --token X.Y \
--discovery-token-ca-cert-hash sha256:XYZ 
#Example Only

• Verify Node Added Successfully (SSH on Master Node)
  • Should have status ready after ~30 seconds

kubectl get nodes

First Deployment and Service

Note: We will perform the deployment via SSH on the Master Node. Below are two ways to deploy the deployment and service; using either YAML or single line commands. YAML allows for easier complex actions while single line commands can be used for simple actions but for this example it will be the same outcome.

• Deploy NGINX (Option A: Simple)

kubectl create deployment nginx --image=nginx

• Deploy NGINX (Option B: YAML)
  • kubectl apply -f nginx.yaml

#nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx

• Testing NGINX

kubectl get deployments

List of running deployments

• Deploy Node Port Service (Option A: Simple)

kubectl create service nodeport nginx --tcp=80:80

• Deploy Node Port Service (Option B: YAML)
  • kubectl apply -f nginxservice.yaml

#nginxservice.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      nodePort: 32000

• Testing Node Port Service

kubectl get services

List of running services

• Testing Nginx (from your laptop or anything with connectivity to the cluster)

curl [IP address of any node (RPi)]:[Port by the Node Port]

Result of curl request.

DNS A Round Robin

As a form of load balancer, I chose to use DNS A Record Round Robbin.

• Added entries for a shared hostname and pointed it to all the IP addresses of the nodes.
  • Note: The CNI creates a VXLAN network that allows all hosts to redirect to the container which hosts the container via the defined Node Port.

A Few Lessons Learnt

• Require sandpits for testing
  • Without proper testing prior to deploying, updating any of the container components might fail. I saw this with the incompatibility of Kubernetes v1.16 with Weaver (hence the v1.15.3 install).
• Auxiliary/Supporting Services requires additional effort
  • Public clouds provide ready to go managed supporting services such as load balancing, DNS, container registries, user authentication etc.
• Additional work arounds required
  • Not just one click deployment, workarounds maybe required depending on the infrastructure.