Kubernetes with K3S

K3S is a lightweight Kubernetes distribution packed into one single binary. This binary provides a comfortable CLI to manage all aspects of the cluster: Create nodes, join nodes, rotate secrets, and even upgrading. This blog post contains a concise overview about the K3S distribution, and its installation and configuration options.

This article originally appeared at my blog admantium.com.

Distribution Overview

K3S is preconfigured with the following components:

• Control plane storage: SQLite instead of ETCD (but etcd3 can be configured)
• CRI: containerd
• CNI: Flanel
• Ingress: Traefik Ingress Controller
• Storage: Custom local storage provider
• Additional features: Helm controller

K3S is lightweight because all control plane components, and all the required external dependencies, are included in the same binary. The K3S installation process will place a single binary file on each node. This file then starts a K3S process with either the role of a k3s server or k3s agent.

Installation Architectures

There are two principle K3S Architecture:

• Single Server Setup: One master node, several worker nodes. The master node stores its data in an embedded SQLite database. The master node is started with the command k3 server. Worker nodes first run k3s join $MASTER_NODE_IP, and then start the agent process with k3s agent.
• Multi Server Setup: Two or more master nodes, several worker nodes. The master nodes use an external database for recording the cluster state. For worker nodes to join the cluster, first, a fixed registration address needs to be configured, and then the worker join exactly to this address.

Installation Process

Single Server Setup

The installation step is delightfully simple:

1. Start the master node by executing the following script:

curl -sfL https://get.k3s.io | sh -

1. Ensure that the master node can be reached by fixed IP address, load-balancer or DNS. Also, retrieve the master node secret token from /var/lib/rancher/k3s/server/node-token

2. Join worker nodes by reusing the same secret and connecting to the configured server:
   

curl -sfL https://get.k3s.io | K3S_URL=$K3S_Server K3S_TOKEN=$K3S_TOKEN sh -

That is all!

Multi Server Setup

As detailed in the official documentation, following steps need to be done:

1. Create an external datastore, such as Postgres, MySQL, MariaDB, or ETCD
2. Install the K3S binary on the server node
3. Start the K3S process in server mode, providing the address of the storage server (like in the following example)

  k3-server \
    --token=$SECRET \
    --datastore-endpoint=$DATASTORE_ENDPOINT_URL

1. One master node needs to have a fixed registration address, like a DNS entry virtual IP, to which the nodes connect: You need to configure one such address, and use this also for connecting to the Kube API of the server
2. On additional server nodes, start the same command as in step 3, using the same token
3. Join worker nodes by running the following command that connects to the fixed registration address:

K3S_TOKEN=$SECRET k3s agent --server $FIXED_REGISTRATION_ADDRESS:6443

Upgrade Process

Because K3S is a single binary, and all Kubernetes components are bundled into it, upgrading is delightfully simple as well. The K3S binary is provided in different release channels. The canonical stable and latest point to the most current Kubernetes versions. And for specific versions, the release channels use the same version number, e.g. 1.21 or 1.22. Simply pass an environment variable that specifies the desired version, as shown in the following example:

curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=v1.22.1 sh -

Another option is to use an Kubernetes operator, a set of powerful abstractions, that plan and execute upgrades as configured in manifests file. See the official documentation on k3s automatic updates.

Customization

K3S allows several customization options, most are passed either as environment variables or flags during the installation, or as flags to the binary that is started on each node. See the server documentation and the agent documentation.

For the Kubernetes components, you have these options:

• Control Plane Storage
  • Embedded SQLite
  • PostgreSQL
  • MySQL
  • MariaDB
  • Etcd
• Container Runtime
  • containerd
  • Docker
• Container Network Interface
  • Flanel
  • Canal
  • Calico

Conclusion

In this article, you learned about the lightweight Kubernetes distribution K3S. You saw a list of its main components, and its two principal architectures, which are one master vs. multi-master with external storage. Furthermore, you learned about the installation and upgrade process, and saw the customization options.