Kubernetes Cluster Installation: K3S

There are several Kubernetes distributions. A remarkable simple on is K3S, a lightweight distribution with a small CPU/memory footprint that can run on anything from cloud vm, bare metal to IOT edge devices, including a Raspberry Pi. This is achieved because K3S comes bundled as a single binary with all K3S components and using an embedded SQL lite database. Read more about K3S in my previous article.

This article is a hands-on tutorial to install a k3s Kubernetes cluster with a single control plane node (controller from here on). We start with the prerequisite setup of the nodes, then install the control plane, and add worker nodes. Finally, we will also see how to upgrade the cluster.

This article originally appeared at my blog admantium.com.

Prerequisites

To follow along, you need this:

• 3 server with a fully installed Linux (preferably Debian or Ubuntu)
• SSH access

My target infrastructure in this tutorial are cloud servers from Hetzner with these specifications:

• 1x CX11 node (1 Intel CPU, 2GB RAM, 40GB SSD)
• 2x CX21 nodes (2 Intel CPU, 4GB RAM, 40GB SSD)

Here is a screenshot of the Hetzner management UI:

Generate SSH Key

Create a new ssh key with the following command:

ssh-keygen -t rsa -b 4096 -C K8S_Trial

Use ssh-copy or a similar mechanism to provide your keys to the target infrastructure.

Install K3S on the Control Plane Node

Connect to the server that is designated as your controller and run the install script with your preferred Kubernetes version.

$> curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=v1.22.5+k3s2 sh -


[INFO]  Finding release for channel v1.22.5+k3s2
[INFO]  Using v1.22.5+k3s2 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.22.5+k3s2/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.22.5+k3s2/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

Once completed, you can see the running K3S process:

tatus k3s
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-08-25 17:50:23 UTC; 9min ago
       Docs: https://k3s.io
    Process: 611 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setu>
    Process: 634 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 657 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 666 (k3s-server)
      Tasks: 64
     Memory: 661.8M
     CGroup: /system.slice/k3s.service
             ├─ 666 /usr/local/bin/k3s server
             ├─ 717 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run>
             ├─1161 /var/lib/rancher/k3s/data/dee5defe4c7598297b43a88507ac754f06017d9466984bd51>
             └─1863 /var/lib/rancher/k3s/data/dee5defe4c7598297b43a88507ac754f06017d9466984bd51>

Install K3S on Worker Nodes

The installation procedure for worker nodes is similarly simple, with one addition: We need to get a unique secret from the controller which is consumed by the workers for the initial connection.

Get the secret as follows:

$> ssh devcon@k3s-controller cat /var/lib/rancher/k3s/server/node-token

Then, access the worker node, and run the following command:

$> curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=v1.22.5+k3s2 K3S_URL=https://135.181.205.14:6443 K3S_TOKEN=$SECRET sh -

[INFO]  Finding release for channel v1.22.5+k3s2
[INFO]  Using v1.22.5+k3s2 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.22.5+k3s2/sha256sum-amd64.txt>
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.22.5+k3s2/k3s>
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO]  systemd: Enabling k3s-agent unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service.
[INFO]  systemd: Starting k3s-agent

As before, check that the service is running correctly:

systemctl status k3s-agent
● k3s-agent.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-08-25 17:50:29 UTC; 8min ago
       Docs: https://k3s.io
    Process: 588 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setu>
    Process: 613 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 645 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 649 (k3s-agent)
      Tasks: 74
     Memory: 340.6M
     CGroup: /system.slice/k3s-agent.service
             ├─ 649 /usr/local/bin/k3s agent
             ├─ 683 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run>
             ├─1267 /var/lib/rancher/k3s/data/dee5defe4c7598297b43a88507ac754f06017d9466984bd51>
             ├─1268 /var/lib/rancher/k3s/data/dee5defe4c7598297b43a88507ac754f06017d9466984bd51>
             ├─1678 /var/lib/rancher/k3s/data/dee5defe4c7598297b43a88507ac754f06017d9466984bd51>
             └─2162 /var/lib/rancher/k3s/data/dee5defe4c7598297b43a88507ac754f06017d9466984bd51>

Access your Kubernetes Cluster

The kubeconfig is located on the controller in the file /etc/rancher/k3s/k3s.yaml. Copy this with SSH to your machine:

$> scp devcon@k3s-controller:/etc/rancher/k3s/k3s.yaml kubeconfig.yaml

... and change the cluster URL to the DNS or IP address of your controller node:

apiVersion: v1
clusters:
  - cluster:
      certificate-authority-data: LS
      server: https://$SERVER_IP:6443

Now we are ready to use kubectl for connecting to the cluster. There is a small caveat. According to the Kubernetes release cycle, the kubectl version supports the current and current +2 minor versions. When managing different clusters with different kubectl version, you cannot rely on only one version installed on your client machine. Therefore, we will use the docker container bitnami/kubectl which offers a version for each Kubernetes major, minor and patch version

Start the container with the following command and mount a folder that contains the kubeconfig file into the container.

$> docker run -d \
  -it \
  --entrypoint=/bin/bash \
  --name k3s \
  --network="host" \
  --mount type=bind,source=/Users/work/development/k8s/rancher-k3s/,target=/k8s \
  bitnami/kubectl:1.22.5

Use the following command to start a shell inside the Docker container, and then you can access your Kubernetes cluster.

$> docker exec -it k3 sh

$> export KUBECONFIG=/k8s/kubeconfig.yaml
$> kubectl get nodes

NAME         STATUS   ROLES                  AGE   VERSION
worker2      Ready    <none>                 111s   v1.22.5+k3s2
controller   Ready    control-plane,master   37m    v1.22.5+k3s2
worker1      Ready    <none>                 70s    v1.22.5+k3s2

Excellent! All nodes are created, the cluster is usable.

Upgrade the Kubernetes Version

Upgrading K3S is a bliss: Because all Kubernetes components are bundled with the k3s binary, you simple exchange the binary to a newer version. In the prior section, Kubernetes version v1.22.5 was used. To upgrade this to v1.23.8, simple execute the following command on the controller node:

$> curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=v1.23.8+k3s2 sh -

[INFO]  Finding release for channel v1.23.8+k3s2
[INFO]  Using v1.23.8+k3s2 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.23.8+k3s2/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.23.8+k3s2/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Skipping /usr/local/bin/kubectl symlink to k3s, already exists
[INFO]  Skipping /usr/local/bin/crictl symlink to k3s, already exists
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, already exists
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

On the worker nodes, the command is slightly different:

curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=v1.23.8+k3s2 K3S_URL=https://135.181.205.14:6443 K3S_TOKEN=$SECRET sh -

Thats it! Using the Docker container from above to see the results:

kubectl --kubeconfig=$KUBECONFIG get nodes

NAME         STATUS   ROLES                  AGE     VERSION

controller   Ready    control-plane,master   45m     v1.23.8+k3s2

worker1      Ready    <none>                 9m26s   v1.23.8+k3s2

worker2      Ready    <none>                 8m45s   v1.22.5+k3s2

Conclusion

The Kubernetes distribution K3S is resource effective and can be used on cloud VMs, bare metal and edge devices like a Raspberry Pi. Delivered as a single binary that contains all the Kubernetes components, installation and setup is a matter of minutes. K3S works out of the box with no additional configuration requires. Delightfully simple is also the upgrade process: Stop the daemon process, install the next new minor version of the binary, and you are done. I'm using K3S to host this blog since about 2 years and can highly recommend it.