DEV Community

Shankar
Shankar

Posted on

From Zero to GitOps: Building a k3s Homelab on a Raspberry Pi with Flux & SOPS

#kubernetes #gitops #raspberrypi #k3s

This post documents the end-to-end process for setting up a k3s Kubernetes cluster on a Raspberry Pi, managing it remotely from a Mac, and deploying applications securely using GitOps with FluxCD and SOPS encryption. We'll cover everything from OS install to deploying encrypted secrets and tackling common troubleshooting hurdles.

1. Initial Pi Setup & OS Installation

This phase covers preparing the Raspberry Pi hardware and operating system.

  1. Install OS: Use the Raspberry Pi Imager to write Raspberry Pi OS (64-BIT) to an SD card.
  2. OS Configuration: In the Imager's advanced settings, pre-configure:
    • Hostname: k3s-node (or your preferred name)
    • Username and Password: e.g., pi-admin
    • Wireless LAN: Your Wi-Fi SSID and password.
  3. Set a Static IP: To ensure a stable connection, set a DHCP Reservation for the Pi in your home router's settings, linking the Pi's MAC address to a specific IP (e.g., 192.168.1.100).

2. Kubernetes (k3s) Installation

We installed k3s, a lightweight Kubernetes distribution, directly onto the Pi's operating system.

  1. Enable Cgroups (Critical Fix): The k3s service will crash on startup without this Linux kernel feature.
    • SSH into the Pi: ssh pi-admin@k3s-node.local
    • Edit the boot config file: sudo nano /boot/firmware/cmdline.txt
    • Add cgroup_memory=1 cgroup_enable=memory to the end of the single line in the file.
    • Save (Ctrl+X, Y, Enter) and reboot: sudo reboot.

  2. Install k3s:

    # On the Pi
curl -sfL [https://get.k3s.io](https://get.k3s.io) | sh -

  3. Verify Service: Ensure the k3s service is stable and running.

    # On the Pi
sudo systemctl status k3s.service

    The output must show Active: active (running).

3. Remote Management from macOS

To manage the Pi's cluster from your Mac, you need to copy its configuration.

  1. Update Kubeconfig File:
    • On the Pi, copy the config: sudo cat /etc/rancher/k3s/k3s.yaml > k3s_config.yaml
    • Edit the file (nano k3s_config.yaml) and change the server address from https://127.0.0.1:6443 to the Pi's static IP (e.g., https://192.168.1.100:6443).

  2. Copy to Mac: From your Mac's terminal, copy the file to your local kubeconfig location. Warning: This overwrites your default config. If you manage multiple clusters, merge this file's contents manually.

    scp pi-admin@k3s-node.local:~/k3s_config.yaml ~/.kube/config

  3. Test Connection:

    kubectl get nodes

    You should see your Pi node (k3s-node) listed.

4. GitOps Setup with Flux & SOPS

This phase automates deployments and configures secret encryption.

  1. Bootstrap Flux: Install Flux on the cluster and configure it to watch your Git repository.

    # On your Mac
# (Ensure GITHUB_USER is set in your env)
flux bootstrap github \
  --owner=$GITHUB_USER \
  --repository=pi-cluster \
  --branch=main \
  --path=./clusters/staging \
  --personal

  2. Generate age Keypair: Create a new keypair for encryption.

    # On your Mac
age-keygen -o age.agekey

    This creates age.agekey (your private key) and shows your public key (starts age1...). Keep the private key safe!

  3. Add Private Key to Cluster: Create a Kubernetes secret in the flux-system namespace containing your private key. This allows Flux's controllers to decrypt files.

    # On your Mac
cat age.agekey | kubectl create secret generic sops-age \
  --namespace=flux-system \
  --from-file=age.agekey=/dev/stdin

  4. Configure SOPS Rules: Create a .sops.yaml file in clusters/staging/ to tell SOPS which public key to use for encrypting files.

    # In clusters/staging/.sops.yaml
creation_rules:
  - path_regex: .*.yaml
    encrypted_regex: ^(data|stringData)$
    age: <PASTE_YOUR_PUBLIC_AGE_KEY_HERE>

  5. Configure Flux for Decryption: Edit clusters/staging/flux-system/kustomization.yaml to tell Flux to use the sops-age secret.

    # In clusters/staging/flux-system/kustomization.yaml
spec:
  # ...
  decryption:
    provider: sops
    secretRef:
      name: sops-age

  6. Commit & Push your new .sops.yaml and modified kustomization.yaml to Git.

5. Deploying Encrypted Secrets via GitOps

This is the process of creating encrypted secret files and adding them to your Git repository for Flux to deploy.

1. : Deploy the Cloudflare Secret

  1. Generate Secret YAML: Create a YAML manifest from your Cloudflare credential file (<tunnel_id>.json).

    kubectl create secret generic tunnel-credentials \
  --from-file=credentials.json=./<tunnel_id>.json \
  --namespace <your-app-namespace> \
  --dry-run=client -o yaml > cloudflare-secret.yaml

  2. Encrypt the File:

    sops --config clusters/staging/.sops.yaml --encrypt --in-place cloudflare-secret.yaml

  3. Move and Rename: Move the encrypted secret into your application's directory (e.g., apps/base/linkding/secret-cloudflare.sops.yaml).

2: Deploy the Linkding Superuser Secret

  1. Generate Secret YAML: Create a secret with the environment variables linkding expects.

    kubectl create secret generic linkding-superuser \
  --from-literal=LD_SUPERUSER_NAME=your-user \
  --from-literal=LD_SUPERUSER_PASSWORD=YourSecurePassword \
  --namespace <your-app-namespace> \
  --dry-run=client -o yaml > secret-superuser.yaml

  2. Encrypt and Move:

    sops --config clusters/staging/.sops.yaml --encrypt --in-place secret-superuser.yaml
mv secret-superuser.yaml apps/base/linkding/secret-superuser.sops.yaml

  3. Update kustomization.yaml: Edit apps/base/linkding/kustomization.yaml to tell Flux to deploy these new secret files.

    resources:
  - namespace.yaml
  - deployment.yaml
  # ... other resources
  - secret-cloudflare.sops.yaml
  - secret-superuser.sops.yaml

  4. Update deployment.yaml: Modify apps/base/linkding/deployment.yaml to use the superuser secret.

    # In deployment.yaml, inside the container spec:
envFrom:
  - secretRef:
      name: linkding-superuser

Final Step: Commit and Reconcile

After adding the files and updating the Kustomizations, commit everything to Git. Flux will automatically sync the changes, decrypt the secrets, and deploy them to your cluster.

git add .
git commit -m "Feat: Add encrypted secrets for Cloudflare and Linkding"
git push origin main
Enter fullscreen mode Exit fullscreen mode

6. Troubleshooting Common Issues

ImagePullBackOff

Symptom: Kubernetes can't download the container image. You see this status when you run kubectl get pods.

  • Cause 1: Wrong Architecture. You're trying to run an amd64 (standard PC/server) image on your arm64 Raspberry Pi.

  • Solution 1: Find a multi-arch image or an arm64/aarch64 specific version. Look for tags like -arm64, -aarch64, or check image descriptions on Docker Hub/GHCR. lscr.io (LinuxServer.io) often provides good multi-arch images. Update the image: tag in your deployment.yaml and git push.

  • Cause 2: Pi Network/DNS Issues. The Pi itself can't reach the container registry.

  • Solution 2: SSH into the Pi.

    1. Test basic connectivity: ping google.com
    2. Test DNS: nslookup ghcr.io
    3. If DNS fails, try setting static DNS servers: edit /etc/dhcpcd.conf (sudo nano /etc/dhcpcd.conf).
    4. Add a line: static domain_name_servers=8.8.8.8 1.1.1.1 (Google/Cloudflare DNS).
    5. Save and reboot (sudo reboot).

Pod Stuck in Pending

Symptom: The pod stays in the Pending state and never starts. Running kubectl describe pod <pod-name> shows an event like failed to bind volume.

  • Cause: The pod is waiting for a PersistentVolumeClaim (PVC), but no suitable PersistentVolume (PV) is available to fulfill it (e.g., wrong size, access mode, storage class, or no PVs exist).
  • Solution (Simple hostPath PV for Homelab): Define a PersistentVolume in your GitOps repo that uses a directory on the Pi's filesystem. Warning: hostPath ties the data to that specific Pi node.

  1. Create a directory on the Pi:

    sudo mkdir -p /mnt/data/my-app-data && sudo chown nobody:nogroup /mnt/data/my-app-data

    (Adjust owner if your app runs as a different user).

  2. Create a pv.yaml manifest in your GitOps repo (e.g., alongside the PVC):

    apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-app-data-pv # Unique name
spec:
  capacity:
    storage: 5Gi # Must be >= PVC request
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce # Must match PVC
  persistentVolumeReclaimPolicy: Retain # Keep data if PV is deleted
  storageClassName: manual # Give it a name
  hostPath:
    path: "/mnt/data/my-app-data" # Path on the Pi node

  3. Add pv.yaml to your Kustomization.

  4. Update your application's PVC spec.storageClassName to manual (or whatever name you chose) so it binds to this PV.

Connection Refused / ServiceUnavailable (from remote kubectl)

Symptom: Running kubectl get nodes from your Mac or remote machine fails with ServiceUnavailable or connection refused.

  • Cause: The k3s service on the Pi is down, restarting, or unstable. This is almost always caused by:
    1. Forgetting the cgroups fix (critical for ks on Raspberry Pi OS).
    2. The Pi is out of resources (memory/CPU).
  • Solution:
    1. SSH into the Pi.
    2. Check the k3s service status: sudo systemctl status k3s.service.
    3. If it's not active (running), check the logs for crash reasons: sudo journalctl -u k3s.service -f.
    4. Confirm /boot/firmware/cmdline.txt includes the cgroup_memory=1 cgroup_enable=memory flags (and reboot if you had to add them).
    5. Check resource usage with htop.

Top comments (0)