DEV Community: Algo7

K3s Update TLS SANs

Algo7 — Sat, 09 Mar 2024 21:02:44 +0000

Introduction

When I was following the official guide of adding an external cluster load balancer for K3s using HAProxy to have some HAs when connecting to the cluster. I ran into a problem: kubectl refuses to connect to the cluster due to some certificate issues:

couldn't get current server API group list: Get "https://balancer_ip:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate is valid for [list of control plan ips and cluster + Kube API server service ip], ::1, not load_balancer_ip

This is due to my load balancer's IP not being in the cluster's API server certificate. During cluster initialization and when adding new control plane node to the cluster, K3s automatically update this certificate to include the new control plane node. However, this is not the case when adding an external cluster load balancer.

Solution 1: Reinitialize the Cluster With --tls-san
Solution 2: Editing K3s Configuration File
- Step 1: Edit the Config File
- Step 2: Delete the Existing Kube API Server Cert
- Step 3: Restart K3s
- Step 4: Inform the DynamicListener About the Change
  - Alternative Method
Worker / Control Plane Node Reconfiguration (Optional)

Solution 1: Reinitialize the Cluster With --tls-san

According to the K3s official documentation, you can pass the --tls-san flag when initializing the cluster:

sh k3s_install.sh --tls-san additional_ip_1 --tls-san additional_ip_1 --cluster-init

However, my cluster is already running and I don't want to reinstall it.

Solution 2: Editing K3s Configuration File

The K3s configuration file is located under /etc/rancher/k3s/config.yaml on all control plane nodes. If the file doesn't exist, you have to create it manually.

Step 1: Edit the Config File

In the config file, you can simply add the following entries:

tls-san:
  - ip_1
  - hostname_1
  // additional entries

This can be done on any control plane node.

Step 2: Delete the Existing Kube API Server Cert

On any control plane node run

sudo kubectl -n kube-system delete secrets/k3s-serving

to delete the existing Kubernetes API server certificate.

This can be done anywhere as long as you can access the cluster's API server.

Step 3: Restart K3s

Restart K3s on the same control plane node:

sudo systemctl restart k3s

Step 4: Inform the DynamicListener About the Change

DynamicListener is a component of K3s that handles automatic updates/renewal of the API server certificate, including when new control plan nodes join the cluster.

According to one of the K3s contributor, Brad Davidson in one of the GitHub issue:

Dynamiclistener adds SANs for any hostname or IP address requested via a HTTP Host header or TLS SNI handshake. It is designed to allow you to add or remove servers without having to manually regenerate the certificate. The downside (as you noted) is that this allows for SAN stuffing.

The documentation about it is not very clear and the official README is still in progress. This is as far as I can go about it.

For each new hostname / IP you added to the config file, run the following command on the control plan node:

curl -k --resolve your_new_hostname_or_ip:6443:127.0.0.1  https://your_new_hostname_or_ip:6443/ping

This method is also mentioned in another GitHub issue comment here.

To confirm that the API server certificate has been updated, run the following command:

kubectl get secret/k3s-serving -n kube-system -o yaml

This can be done anywhere as long as you have access to the cluster's API server.

In the metadata.annotations part of the output, you should see your newly added hostname/ip as one of the annotations:

listener.cattle.io/cn-ur_hostnameor__ip: ur_hostname_or_ip

Alternative Method

I found one Medium post here that uses another method to update the DynamicListener.

After you delete the existing API server certificate on a control plane node using

sudo kubectl -n kube-system delete secrets/k3s-serving

You can move or delete a file called dynamic-cert.json located under /var/lib/rancher/k3s/server/tls on the same control plane node

sudo mv /var/lib/rancher/k3s/server/tls/dynamic-cert.json /tmp/dynamic-cert.json

BEFORE restarting K3s on the same control plane node.

I have tested both methods and they all worked.

Worker / Control Plane Node Reconfiguration (Optional)

Now we have to update our worker / control plane nodes to use the new endpoint.

The fastest way is to update the K3s systemd service file.

You will have to do this on each node that you want to reconfigure.

For control plane nodes the file is located at

/etc/systemd/system/k3s.service

For worker nodes the file is located at

/etc/systemd/system/k3s-agent.service

Update the value of the --server flag in the file to use the new hostname / IP of the balancer.

After making the changes, run sudo systemctl daemon-reload to reload the changes.

Finally run sudo systemctl restart k3s to restart the K3s daemon for the changes to take effect.

Install K3s on Proxmox Using Ansible

Algo7 — Tue, 30 Jan 2024 12:50:23 +0000

Introduction

This article will guide you through how to install K3s with Flannel VXLAN backend on Proxmox using Ansible. It is assumed that you have basic knowledge about Kubernetes, Proxmox, SSH, and Ansible.

This is my first post on dev.to, any constructive feedback is welcomed.

Introduction
Table Of Contents
Prerequisites
Proxmox Firewall Configuration
- Creating Proxmox Security Group
- Applying the Security Groups to your VMs
Ansible Setup
- Update your SSH config
- Directory Structure
- Ansible Config File
- Ansible Inventory
- The Enrollment Token
- The K3s Ansible Role
- The Playbook
Run the Playbook
Cluster Access
Resources

Prerequisites

Promox installed and running
Ansible installed on your local machine
At least 4 VMs (3 control plane nodes + 1 worker node)
- OS: Debian-based OSes, preferably the latest Ubuntu LTS
- SSH installed and configured using Public Key Auth
kubectl installed locally
SSH installed locally

Proxmox Firewall Configuration

When you create VMs on Proxmox, the firewall is disabled by default. If you want to use the VM firewall, please make sure you opened the required ports.

Below are the inbound rules for K3s nodes from the official documentation:

For more information: https://docs.k3s.io/installation/requirements#inbound-rules-for-k3s-nodes

We don't need all of these ports opened because we are not using the Flannel Wireguard backend and Spegel registry.

Creating Proxmox Security Group

As we will be applying the same firewall rules to multiple VMs, it will be easier if we create a security group in Proxmox and apply the security group to all the target VMs.

We will be creating 2 Security Groups in Proxmox:

k3s: Ports that need to be opened on all nodes
k3s_server: Ports that only need to be opened on the control plane nodes

To create a security group in Proxmox:

Login to the Proxmox UI
Go to Datacenter on the left menu
Go to Firewall and then Security Group
Click on Create
Select the security group you just created and click on Add to start adding rules

For the k3s security group:

TCP 10250 for Kubelet metrics
UDP 8472 for Flannel VXLAN
TCP 22 for SSH

For the k3s_server security group:

TCP 6443 for K3s supervisor and Kubernetes API Server
TCP 2479 for HA with embedded etcd
TCP 2380 for HA with embedded etcd

Applying the Security Groups to your VMs

Click on your VM on from the left menu
Select Firewall
Click on the Insert: Security Group button
Select the security groups we just created and enable them

Ansible Setup

Update your SSH config

To streamline our process of working with Ansible, it is recommended that you add the target hosts to your SSH config so we can reference them using hostnames in the Ansible inventory file. I am using Ubuntu on my local machine, so the SSH config is located at ~/.ssh/config. The same should apply to all Linux/Unix OSes.

Example configuration

# Control plane 1
Host k3s-m1
  User ubuntu
  HostName 10.0.0.30
  IdentityFile ~/.ssh/your_pk

# Control plane 2
Host k3s-m2
  User ubuntu
  HostName 10.0.0.31
  IdentityFile ~/.ssh/your_pk

# Control plane 3
Host k3s-m3
  User ubuntu
  HostName 10.0.0.32
  IdentityFile ~/.ssh/your_pk

# Worker node 1
Host k3s-w1
  User ubuntu
  HostName 10.0.0.33
  IdentityFile ~/.ssh/your_pk

# Add more if needed

Directory Structure

.
├── ansible.cfg # project specific Ansible configuration
├── inventory
│   └── inventories.ini # Inventory information
├── k3s.yml # The playbook to install K3s
└── roles
    └── k3s
        ├── tasks
        │   └── main.yml # Ansible role to install K3s
        └── vars
            └── k3s-secrets.yml # K3s enrollment token

Ansible Config File

Path: project_root/ansible.cfg

We will create a custom Ansible configuration file named ansible.cfg at the root of the project directory, which will reference the inventory file.

Example configuration:

[defaults]
inventory = ./inventory/inventories.ini
# If you haven't connected to the K3s VMs before and they are not in
# your SSH `know_hosts` file, you will have to uncomment the
# following option for the playbook to not thrown an error:
# host_key_checking = False

Ansible Inventory

Path: project_root/inventory/inventories.ini

We will organize the target VMs using an Ansible inventory file.

# Initial master node setup for bootstrapping the K3S cluster.
# 'node_type' is used in the Ansible roles to identify and execute specific tasks for this node (see the role section).
[k3s_initial_master]
# k3s-m1 is the hostname we defined in our SSH config
k3s-m1 node_type=k3s_initial_master

# Additional master nodes for the K3S cluster.
[k3s_masters]
k3s-m2 node_type=k3s_master
k3s-m3 node_type=k3s_master
# Additional workers...

# Worker nodes for running containerized applications.
[k3s_workers]
k3s-w1 node_type=k3s_worker
k3s-w2 node_type=k3s_worker
k3s-w3 node_type=k3s_worker
# Additional workers...

# Group definition for simplified playbook targeting.
[k3s:children]
k3s_initial_master
k3s_masters
k3s_workers

The Enrollment Token

Path: project_root/roles/k3s/vars/k3s-secrets.yml

You can optionally encrypt the file using the ansible-vault encrypt path_to_file command

---
# You have to create a token yourself
k3s_token: "your_token"

The K3s Ansible Role

Path: project_root/roles/k3s/tasks/main.yml

# code: language=ansible
---
- name: Install K3S Requirements
  ansible.builtin.apt:
    update_cache: true
    pkg:
      - policycoreutils
    state: present

# To make sure the the role is idempotent. The tasks after this will only be executed if K3S hasn't been installed already.
- name: Check if K3S is already installed
  ansible.builtin.shell:
    cmd: 'test -f /usr/local/bin/k3s'
  register: k3s_installed
  failed_when: false

- name: Download K3s installation script
  ansible.builtin.uri:
    url: 'https://get.k3s.io'
    method: GET
    return_content: yes
    dest: '/tmp/k3s_install.sh'
  when: k3s_installed.rc != 0

- name: Import K3S Token
  ansible.builtin.include_vars:
    file: k3s-secrets.yml.vault
  when: k3s_installed.rc != 0

# Note that the node_type variable is set in the inventory file
- name: Execute K3s installation script [Initial Master Node]
  ansible.builtin.shell:
    cmd: 'sh /tmp/k3s_install.sh --token {{ k3s_token }} --disable=traefik --flannel-backend=vxlan --cluster-init'
  vars:
    k3s_token: '{{ k3s_token }}'
  args:
    executable: /bin/bash
  when: node_type | default('undefined') == 'k3s_initial_master' and k3s_installed.rc != 0

- name: Execute K3s installation script [Master Nodes]
  ansible.builtin.shell:
    cmd: 'sh /tmp/k3s_install.sh --token {{ k3s_token }} --disable=traefik --flannel-backend=vxlan --server https://{{ hostvars["k3s-m1"]["ansible_default_ipv4"]["address"] }}:6443'
  vars:
    k3s_token: '{{ k3s_token }}'
  args:
    executable: /bin/bash
  when: node_type | default('undefined') == 'k3s_master' and k3s_installed.rc != 0

- name: Execute K3s installation script [Worker Nodes]
  ansible.builtin.shell:
    cmd: 'sh /tmp/k3s_install.sh agent --token {{ k3s_token }} --server https://{{ hostvars["k3s-m1"]["ansible_default_ipv4"]["address"] }}:6443'
  vars:
    k3s_token: '{{ k3s_token }}'
  args:
    executable: /bin/bash
  when: node_type | default('undefined') == 'k3s_worker' and k3s_installed.rc != 0

The Playbook

Path: project_root/k3s.yml

# code: language=ansible
# K3S Ansible Playbook
---
- name: K3S
  hosts: k3s
  gather_facts: true
  roles:
    - role: k3s
      become: true

Run the Playbook

In your terminal, run ansible-playbook k3s.yml. If you have encrypted the k3s-secret.yml then you should run ansible-playbook k3s.yml --ask-vault-pass and enter the password.

Cluster Access

After the playbook has finished running. You can obtain the content of the cluster's kubeconfig by SSH into one of the master node and reading the content of /etc/rancher/k3s/k3s.yaml

More information: https://docs.k3s.io/cluster-access

To use the config locally, please remember to change the server: https://127.0.0.1:6443 property in the file to point to one of the master node.

Once you have copied the kubeconfig to your local machine, you can run kubectl get ns to list all the namespaces and test the connection. If you have multiple clusters locally, make sure you are selecting the the correct context.

Resources

The playbook and role configuration can be found here on my GitHub repo: https://github.com/algo7/homelab_ansible/tree/main/roles/k3s

DEV Community: Algo7

K3s Update TLS SANs

Introduction

Table of Contents

Solution 1: Reinitialize the Cluster With --tls-san

Solution 2: Editing K3s Configuration File

Step 1: Edit the Config File

Step 2: Delete the Existing Kube API Server Cert

Step 3: Restart K3s

Step 4: Inform the DynamicListener About the Change

Alternative Method

Worker / Control Plane Node Reconfiguration (Optional)

Install K3s on Proxmox Using Ansible

Introduction

Table of Contents

Prerequisites

Proxmox Firewall Configuration

Creating Proxmox Security Group

Applying the Security Groups to your VMs

Ansible Setup

Update your SSH config

Directory Structure

Ansible Config File

Ansible Inventory

The Enrollment Token

The K3s Ansible Role

The Playbook

Run the Playbook

Cluster Access

Resources