DEV Community

Shankar
Shankar

Posted on

Troubleshooting k3s on Raspberry Pi (Fixing the Auto-Restart Crash Loop)

#kubernetes #k3s #raspberrypi #homelab

So you've installed k3s (the lightweight Kubernetes distribution) on your Raspberry Pi, but your kubectl commands from your main computer are failing with "connection refused"? You SSH into the Pi, check the service status (sudo systemctl status k3s.service), and see it stuck in activating (auto-restart)?

You're likely facing a very common issue, especially on Raspberry Pi OS. Let's diagnose it and fix it step-by-step.

The Symptoms

You'll typically see two related problems:

  1. On your Raspberry Pi: The k3s Kubernetes service itself is crashing and getting stuck in a restart loop.
    • sudo systemctl status k3s.service shows Active: activating (auto-restart) or mentions code=exited, status=1/FAILURE.
    • Running kubectl get nodes on the Pi (with sudo k3s kubectl) might intermittently work or show The connection to the server 127.0.0.1:6443 was refused.
  2. On your Remote Machine (e.g., Mac/Linux/Windows): Your kubectl commands targeting the Pi fail consistently with connection errors (like connection refused or timeouts) because the k3s API server on the Pi isn't reliably available.

The core issue we need to solve is on the Raspberry Pi: why is k3s crashing?

The Problem: k3s is Crashing

The systemctl status k3s.service output tells the story:

  • Active: activating (auto-restart): The service manager (systemd) is trying to start k3s, it fails, and systemd automatically tries again, repeatedly.
  • (code=exited, status=1/FAILURE): This confirms the main k3s process crashed with an error.

The connection refused errors happen because kubectl tries to talk to the k3s API server while it's down during one of these crashes or restarts.

The Most Likely Cause on Raspberry Pi

For Raspberry Pi setups, this crash-restart loop is almost always due to missing Linux kernel features required by k3s, specifically related to Control Groups (cgroups) for memory management. The k3s installation script often warns about this, but it's an easy step to miss.

The Fix: Enable Cgroups and Reinstall k3s

We'll ensure the kernel is configured correctly and then perform a clean re-installation of k3s to fix any potentially corrupted state from the failed starts.

1. Enable Cgroups on the Raspberry Pi

  1. SSH into your Raspberry Pi using its hostname or IP address:

    ssh <your_pi_user>@<your_pi_hostname_or_ip>
# e.g., ssh pi-admin@k3s-node.local

  2. Edit the boot configuration file using sudo permissions:

    sudo nano /boot/firmware/cmdline.txt

  3. This file contains a single, long line of text. Use your arrow keys to navigate to the very end of that line.

  4. Add a single space, and then paste the following text:

    cgroup_memory=1 cgroup_enable=memory

    (Make absolutely sure the entire file content remains on one single line!)

  5. Save the file by pressing Ctrl + X, then Y, then Enter.

2. Cleanly Uninstall k3s

Let's remove the current (likely broken) installation.

  1. Run the official uninstall script (this stops the service and removes k3s files):

    # Run this command if it exists
/usr/local/bin/k3s-uninstall.sh

# If the above gives "command not found", try this one:
# /usr/local/bin/k3s-agent-uninstall.sh

  2. Now, reboot the Pi to apply the kernel changes from Step 1:

    sudo reboot

3. Reinstall k3s and Verify

  1. Wait for the Pi to reboot, then SSH back in.

  2. Run the installation script again:

    curl -sfL [https://get.k3s.io](https://get.k3s.io) | sh -

  3. Give it a minute to start up, then check the service status again:

    sudo systemctl status k3s.service

    You should now see the glorious green text: Active: active (running). If it's stable and running, you've fixed the main issue!

Final Step: Update Your Remote Kubeconfig

Because k3s was reinstalled, it has generated new security certificates. The old configuration file on your Mac is now invalid. You need to copy the new one over.

  1. On the Pi: Copy the new config to your home directory:

    sudo cat /etc/rancher/k3s/k3s.yaml > $HOME/k3s_config.yaml

  2. On the Pi: Edit the copied file (nano $HOME/k3s_config.yaml) and change the server: address from https://127.0.0.1:6443 to use your Pi's static IP address (e.g., https://192.168.1.100:6443). Save and exit.

  3. On your Mac: Use scp to copy the updated file, replacing your old config. (Remember: back up ~/.kube/config first if you have other cluster contexts in it!)

    # On your Mac - use your Pi's user/hostname/IP
scp <your_pi_user>@<your_pi_hostname_or_ip>:~/k3s_config.yaml ~/.kube/config

  4. Test Again:

    # On your Mac
kubectl get nodes

    Your kubectl commands should now connect successfully and consistently!

Top comments (0)