Let's Make a k3s Cluster with Raspberry Pi and Cloudflare Tunnel

This time I am going to write about how I made my k3s cluster using Raspberry Pi that are accessible from the internet.

I sometimes have this feeling that I want to expose my hobby projects and get some feedbacks from the internet. Leveraging cloud services like AWS should be still a go-to option for most of developers for their hobby projects. But this time I decided to do it in a different way - k3s.

For those who don't know k3s, it's a light weight version of Kubernetes suitable for IoT & Edge devices. One of the advantages of k3s is it's easier to install and make it up and running.

I have three Raspberry Pi devices on my desk (I bought them before Covid 19 pandemic so I don't remember the model - probably Pi 4 Model B (4GB)?). To be honest I have built a k3s cluster using Raspberry Pi devices years ago. But now I relocated myself and the current setup in my house, my ISP doesn't allow me to expose my machine to the internet. So this whole idea seems difficult...

And I found Cloudflare Tunnel. Simply put it's a reverse proxy server that can send HTTPS traffic to servers in a private network. The great thing about Cloudflare Tunnel is I can use it for free.

Honestly I thought only one cloudflared instance can connect to the tunnel before writing this article. But it turns out Cloudflare Tunnel supports multiple cloudflared instances so that you can avoid having single point of failure😯

Without further ado, let's get started!

Prerequisites

• Cloudflare account.
• Router with LAN cable slports. While this is not necessary as WiFi connection should works fine. But it's preferable for a reason I'll mention later. My ISP has provided me a modem bundled with a router which has 4 LAN cable ports that's good for me.

Set Up Raspberry Pi

The very first step for this project is to have your Raspberry Pi up and running. This also includes SSH set up. If you already have a running one (or more) you can skip this step.
One thing I want to note here is that I personally recommend you to not use WiFi connection, instead utilize wired connection which is more reliable and fast. I occasionally experienced unexpected disconnection and had to reboot my k3s when I had run MetalLB on the cluster, which we won't use it in this project though.

Here is how I setup LAN and SSH connection during OS write:

1. Download Raspberry Pi Imager and install it on your machine.
2. Insert a SD card that you want to write the OS image on.
3. Open Raspberry Pi Imager.
4. Choose device, OS (Raspberry Pi OS), and Storage (your SD card). DO NOT click NEXT for now.

1. Open Advanced Options window (Ctrl+Shift+X, or Cmd+Shift+X for macOS).
   • In GENERAL tab, change hostname, username/password as you like. Uncheck Configure wireless LAN.
   • In SERVICES tab, check Enable SSH and select Allow public-key authentication only
   • Hit save.

1. Back to the main window, hit NEXT and proceed writing an OS image to the SD card.

Once it's done. Insert it to your Raspberry Pi device, connect an Ethernet cable to it, and turn it on. Hopefully your router will detect your tiny device and assign a new IP address to it. If possible you should fix the IP address so that the router will reassign the same IP address to your device when it gets rebooted. Anyway check the newly assigned IP address as we need it for the next instruction.

Now, write down the following configuration in ~/.ssh/config:

Host <device name>
    HostName <IP address for the device>
    User <username>
    IdentityFile <path to your private key>

After that you should be able to SSH to your Raspberry Pi device with the following command:

ssh <device name>

Run Nginx For Testing Purpose

This section is not mandatory. But I like to check if Cloudflare Tunnel works before dealing with k3s.

sudo apt install nginx -y  # Install Nginx
sudo systemctl start nginx # Start Nginx

You should be able to access to the web server from your PC now (in my case, Mac).

DNS Configuration

We need to let Cloudflare Tunnel control your domain that you want to use. If you have one on Cloudflare, you can use it. If you have one on other DNS registrar like me, you need to transfer your domain to Cloudflare.

In my case I chose to use one on my Squarespace account (formally, Google Domain). Here are the instructions to use my domain name on Cloudflare (I believe it's almost the same for all other DNS registrar):

1. Go to Cloudflare dashboard/Domain Registration/Transfer Domains.
2. Initiate a new transfer.
3. Cloudflare will ask you to change the name servers for your domain. So change it.
4. After a few minutes your domain will show up and be available on Cloudflare.

We don't finish the transfer at this point. But it seems like you can use the domain name for Cloudflare Tunnel and don't have to proceed further procedures. So let me skip that.

Create a Cloudflare Tunnel

Next, head to Cloudflare's dashboard (Zero Trust/ Networks /Tunnels) and create a new tunnel.
You should be able to choose your domain name set up in the previous section.

Then you will be asked to install claredflared on your device. Select Debian for OS, and 64-bit for an architecture, then follow the instructions in the page to install cloudflared on your Raspberry Pi device. Once it's done, hit Next.

In the Public Hostnames tab, enter public hostname info to the form. In the Service section, Select HTTP and localhost (just localhost, with no custom port number for now) Your info should look like below:

Hit Save tunnel.

Run Cloudflared

Now let's SSH your Raspberry Pi and run the following command:

cloudflared login

You will be asked to access URL via a web browser. If success you should see something like below in your Cloudflare dashboard:

Now we are ready. From your PC go to the public URL you registered. You should be able to see the Nginx's default index.html!

Set Up k3s Server

Finally, it's time to deal with our Kubernetes. Quick Start Guide

Let's SSH your Raspberry Pi device. One important thing - by default cgroup is not enabled on Raspberry Pi. So before installing k3s you need to append the following line to /boot/firmware/cmdline.txt.

cgroup_memory=1 cgroup_enable=memory

Also, it's not recommended to have memory swapping enabled:

sudo swapoff --all
sudo systemctl stop dphys-swapfile.service
sudo systemctl disable dphys-swapfile.service

Let's reboot the device now.

sudo reboot

Once it's up again, SSH the device and install k3s.

curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" sh -

When it's done. Check to see if your Kubernetes master is working.

kubectl get pods -A

Local kubectl Set Up

It should be useful to kubectl from your local PC to the Raspberry Pi.

First install kubectl on your PC. Here is an example command for macOS:

brew install kubectl

And in order to kubectl to remote k3s cluster, you need to copy /etc/rancher/k3s/k3s.yaml on the k3s master node and place it to your PC (on macOS, the default kubectl config file is ~/.kube/config). Once it's done, open the yaml file and find loopback address and replace it with the IP address of your Raspberry Pi.

Or, you can run the following command:

NODE_IP_ADDRESS=<IP address of you Raspberry Pi>
NODE_NAME=<name of your Raspberry Pi specified in ~/.ssh/config>

# Get config file from the device
scp $NODE_NAME:/etc/rancher/k3s/k3s.yaml ~/.kube/config

# Replace loopback IP address
sed -i '' "s/127\.0\.0\.1/$NODE_IP_ADDRESS/g" ~/.kube/config

Now check if you can get k3s information from your local PC.

kubectl get pods -A

Create a Pod For Testing Purpose

Let's create a new deployment that includes nginx container.
Create a new folder (say test-app) and inside it, create the following yaml file:

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-nginx-deployment
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: test-nginx
  template:
    metadata:
      namespace: default
      labels:
        app: test-nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.19.2-alpine
          ports:
            - containerPort: 80

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: test-nginx-service
  namespace: default
spec:
  selector:
    app: test-nginx
  ports:
    - protocol: TCP
      port: 80

ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-nginx-ingress
spec:
  ingressClassName: traefik
  rules:
    - host: hello.0o0o.app
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: test-nginx-service
                port:
                  number: 80

Before creating the deployment above, stop you local Nginx server that we run for the test.

On your local PC, run the command below:

kubectl apply -f test-app

And finally, head over to your URL. You should see nginx's default html page again!

Phew! That's a long way.

For now we still have one k3d node in our cluster. In the next article (hopefully) I am going to write how to add other node to the cluster.

Thanks for reading ✌️