Raspberry Pi K3s Cluster

The last time I played with the Raspberry Pi, I used Ansible to configure docker and some services. However, I still wasn't happy with the solution: my RPi 2 was slow to run the Ansible configuration, I couldn't run much on it, and the setup couldn't recover from any errors without manual intervention.

Now I've upgraded my hardware and certainly over complicated the software for a small home server, but now I got metrics =).

Hardware

Since I use Kubernetes daily, I decided to upgrade my home setup to a k3s cluster. I've started with three nodes:

• RPi 2 model B
• RPi 3
• RPi 4 4GB ram

I added "so much" hardware because my objective is to monitor the cluster using Prometheus and Grafana.

The oldest node, RPi 2 model B, got decommissioned in the process. It wasn't consistent and had issues joining the cluster. So I replaced it with an RPi4 to run the cluster with HA.

OS bootstrapping

The first problem I didn't want to deal with manually was configuring the ssh on the RPis. In the previous process, I had one playbook to configure the ssh and user on Raspberian and another to configure everything else. This time, though, the node will boot with password login disabled and with my public keys automatically added to the host. To achieve this, I used Raspberian Firstboot with a custom script to set up the ssh configuration and hostname.

The second step is putting the Rapberian Firstboot image on the SD card with the firstboot.sh and bootstrap.env in the boot dir. After the node boots up, we can proceed directly to the Ansible configuration.

Ansible

Ansible is responsible for the initial setup of the nodes. After that, the playbook optimizes the RPi to run headless, configures k3s, and sets my local kubectl context with the new cluster. Since ArgoCD manages all k3s resources, Ansible also installs it in the cluster. Argo manages itself from here on.

ArgoCD

Why ArgoCD for such a small setup?

In the previous setup, I had to expose the ssh service on the internet to play with the cluster outside the LAN. I also had to run Ansible every time I modified the services. Because my RPi was slow, it was boring to wait several minutes for simple modifications. With GitOps on ArgoCD, I don't need direct access to the nodes to configure the services.

There was a risk that I would exhaust the cluster resources with only the maintenance services, but there are still around 60% free resources after adding:

• ArgoCD
• Prometheus
• Grafana
• External DNS
• Cert-Manager
• Kong

ArgoCD project and application bootstrapping

To bootstrap the ArgoCD configuration, I used the app of apps pattern. This configuration is Ansible's responsibility: it applies to k3s the initial set up. Service and infrastructure applications are separated to give fewer permissions to the non-infrastructure service configuration.

Secrets

My first approach to keeping secrets versioned was using Sealed Secrets. However, I gave up on the sealed secrets because I rebuilt the cluster frequently and always lost the encryption keys since I didn't care to make backups. I found it a lot easier to keep an encrypted secrets.yaml file in the repository. Every time I spin up a new cluster in this workflow, I have to decrypt the file and apply it to the new cluster.
Although it is encrypted, I don't recommend keeping the file in a public Github repository. I'm doing this to avoid losing the secrets while I work out a better flow for personal projects.