DEV Community: Achyuta Das

HA K8s cluster using kube-vip

Achyuta Das — Fri, 09 Jan 2026 15:18:20 +0000

Overview

A stacked HA cluster is a topology where the distributed data storage cluster provided by etcd is stacked on top of the cluster formed by the nodes managed by kubeadm that run control plane components.

Each control plane node runs an instance of the kube-apiserver, kube-scheduler, and kube-controller-manager. The kube-apiserver is exposed to worker nodes using a load balancer.

Each control plane node creates a local etcd member and this etcd member communicates only with the kube-apiserver of this node. The same applies to the local kube-controller-manager and kube-scheduler instances.

This topology couples the control planes and etcd members on the same nodes. It is simpler to set up than a cluster with external etcd nodes, and simpler to manage for replication.

Here's what happens in a 3-node stacked cluster:
Each control plane node runs:

etcd member
kube-apiserver, scheduler, controller-manager

So, you have:

3 etcd members → quorum = 2
3 API servers → load balanced (can handle 1 down)

If one node fails: You still have:

2 etcd members → quorum maintained
2 control plane instances → still available

This is the default topology deployed by kubeadm. A local etcd member is created automatically on control plane nodes when using kubeadm init and kubeadm join --control-plane

Assumptions: You have done cluster bootstrapping using kubeadm before as this document won’t cover everything in detail.

Kubernetes Static Pods

Static Pods are Kubernetes Pods that are run by the kubelet on a single node and are not managed by the Kubernetes cluster itself. This means that whilst the Pod can appear within Kubernetes, it can't make use of a variety of Kubernetes functionality (such as the Kubernetes token or ConfigMap resources). The static Pod approach is primarily required for kubeadm as this is due to the sequence of actions performed by kubeadm. Ideally, we want kube-vip to be part of the Kubernetes cluster, but for various bits of functionality we also need kube-vip to provide a HA virtual IP as part of the installation.

Setting up the infrastructure

Prerequisite:

containerd is installed on the node.
kubeadm, kubelet, kubectl are install on the node.

To set up kube-vip for Kubernetes High Availability (HA) with 3 master nodes and a Virtual IP (VIP), follow this structured approach:

Masters: 10.238.40.162, 10.238.40.163, 10.238.40.164
VIP: 10.238.40.166

For this validation, I have used Static Pods way of setting up kube-vip on the cluster.

Cluster bootstrap

Generate static pod manifest

export VIP=10.238.40.166
export INTERFACE=enp19s0

# at the time of writing this doc the latest version if v0.9.2
KVVERSION=$(curl -sL https://api.github.com/repos/kube-vip/kube-vip/releases | jq -r ".[0].name")
# we will be using kube-vip container to generate the manifest
alias kube-vip="ctr image pull ghcr.io/kube-vip/kube-vip:$KVVERSION; ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:$KVVERSION vip /kube-vip"

mkdir -p /etc/kubernetes/manifests

kube-vip manifest pod \
    --interface $INTERFACE \
    --address $VIP \
    --controlplane \
    --services \
    --arp \
    --leaderElection | tee /etc/kubernetes/manifests/kube-vip.yaml

Issues during getting a lease for leader election


E0710 19:09:28.208775       1 leaderelection.go:332] error retrieving resource lock kube-system/plndr-cp-lock: leases.coordination.k8s.io "plndr-cp-lock" is forbidden: User "kubernetes-admin" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-system"

There is an open issue with kube-vip needed super-admin.conf access to boot up. This is the case from kubeadm v1.29.x onwards.

kubeadm init is doing the following:

writes an admin.conf that has no binding to cluster-admin yet
writes a super-admin.conf that has system:masters (super user)
create cluster-admin binding using the super-admin

If kube-vip needs permission during setup it can, either:

wait for admin.conf to receive permissions (not possible AFAIK, given the bootstrap sequence)
use super-admin.conf during bootstrap and then moving to admin.conf
Modify the static pod manifest to use the super-admin.conf

sed -i 's#path: /etc/kubernetes/admin.conf#path: /etc/kubernetes/super-admin.conf#' \
          /etc/kubernetes/manifests/kube-vip.yaml

Initialize the first master node; Use the VIP as the cluster endpoint while running the kubeadm init

sudo kubeadm init  --control-plane-endpoint="10.238.40.166:6443"  --upload-certs  --apiserver-cert-extra-sans "10.238.40.166,10.238.40.164,10.238.40.163,10.238.40.162,10.96.0.1,127.0.0.1,0.0.0.0"  --apiserver-advertise-address 10.238.40.162 -v=5

Modify the static pod manifest to use the admin.conf. Once the first node is initialized. update the static pod manifest which will restart the kube-vip pod. The access to the cluster will be lose for couple of minutes.

sed -i 's#path: /etc/kubernetes/super-admin.conf#path: /etc/kubernetes/admin.conf#' \
          /etc/kubernetes/manifests/kube-vip.yaml

Copy the static pod manifest to other master nodes before adding them to the control plane

scp /etc/kubernetes/manifests/kube-vip.yaml root@10.238.40.163:/etc/kubernetes/manifests
scp /etc/kubernetes/manifests/kube-vip.yaml root@10.238.40.164:/etc/kubernetes/manifests

Run the control plane node join command (output of the kubeadm init) on the other master nodes.

kubeadm join 10.238.40.166:8443 --token <> \
        --discovery-token-ca-cert-hash sha256:<> \
        --control-plane --certificate-key <>

Conclusion

Congratulations! You have successfully deployed a highly available Kubernetes cluster using a stacked etcd topology with kube-vip.

HA K8s cluster using Keepalived and HAProxy

Achyuta Das — Fri, 09 Jan 2026 14:59:10 +0000

Overview

Each control plane node runs an instance of the kube-apiserver, kube-scheduler, and kube-controller-manager. The kube-apiserver is exposed to worker nodes using a load balancer.

Each control plane node creates a local etcd member and this etcd member communicates only with the kube-apiserver of this node. The same applies to the local kube-controller-manager and kube-scheduler instances.

This topology couples the control planes and etcd members on the same nodes. It is simpler to set up than a cluster with external etcd nodes, and simpler to manage for replication.

Here's what happens in a 3-node stacked cluster:

Each control plane node runs:

etcd member
kube-apiserver, scheduler, controller-manager

So, you have:

3 etcd members → quorum = 2
3 API servers → load balanced (can handle 1 down)

If one node fails: You still have:

2 etcd members → quorum maintained
2 control plane instances → still available

This is the default topology deployed by kubeadm. A local etcd member is created automatically on control plane nodes when using kubeadm init and kubeadm join --control-plane

Assumptions: You have done cluster bootstrapping using kubeadm before as this document won’t cover everything in detail.

Setting up the machines

To set up HAProxy + Keepalived for Kubernetes High Availability (HA) with 3 master nodes and a Virtual IP (VIP), follow this structured approach:

Masters: 10.238.40.162, 10.238.40.163, 10.238.40.164
VIP: 10.238.40.166

Install HAProxy + Keepalived on all 3 Masters

sudo apt update 
sudo apt install -y haproxy keepalived

HAProxy Configuration Edit /etc/haproxy/haproxy.cfg on all 3 master nodes:

global
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms
    option httplog
    option dontlognull

frontend kubernetes-apiserver
    bind *:8443
    mode tcp
    option tcplog
    default_backend kubernetes-apiserver

backend kubernetes-apiserver
    mode tcp
    balance roundrobin
    option tcp-check
    server master1 10.238.40.162:6443 check fall 3 rise 2
    server master2 10.238.40.163:6443 check fall 3 rise 2
    server master3 10.238.40.164:6443 check fall 3 rise 2

Keepalived Configuration Only one node at a time will "own" the VIP (managed by Keepalived), but config is present on all. Edit /etc/keepalived/keepalived.conf on each master nodes:

Note: Change the priority value for each node:

Master1: priority 110 (MASTER)
Master2: priority 100 (BACKUP)
Master3: priority 90 (BACKUP)

global_defs {
    router_id LVS_DEVEL
    script_user root
    enable_script_security
}

vrrp_script chk_haproxy {
    script "/bin/curl -f http://localhost:6443/healthz || exit 1"
    interval 2
    weight -2
    fall 3
    rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface enp19s0
    virtual_router_id 51
    priority 110
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass k8s-ha-cluster
    }
    virtual_ipaddress {
        10.238.40.166/24
    }
    track_script {
        chk_haproxy
    }
}

Restart HAProxy and Keepalived

sudo systemctl restart haproxy keepalived
sudo systemctl enable haproxy keepalived

Validate the VIP appears on one node

ip addr show | grep 10.238.40.166

Check service status

sudo systemctl status haproxy
sudo systemctl status keepalived

Bootstrap the cluster

Create a kubeadm-config.yaml file on the first master node Make sure to use the VIP as the control plane endpoint, and include it in the apiServer.certSANs

Important: Change the advertiseAddress field in InitConfiguration to match each master node's IP address.

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.32.6
apiServer:
  certSANs:
    - "10.238.40.166"      # VIP
    - "127.0.0.1"           # Localhost
    - "0.0.0.0"             # Wildcard
    - "10.96.0.1"           # Kubernetes service IP
    - "10.238.40.162"
    - "10.238.40.163"
    - "10.238.40.164"
  extraArgs:
    authorization-mode: Node,RBAC
certificatesDir: /etc/kubernetes/pki
clusterName: pcai
controlPlaneEndpoint: "10.238.40.166:8443"
controllerManager:
  extraArgs:
    bind-address: 0.0.0.0
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
networking:
  dnsDomain: cluster.local
  podSubnet: "172.20.0.0/16"
  serviceSubnet: "172.30.0.0/16"
scheduler:
  extraArgs:
    bind-address: 0.0.0.0
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: "10.238.40.162"
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd

Initialize the cluster

kubeadm init --upload-certs --config kubeadm-config.yaml -v=5

Note: Save the output! It contains the join commands for control plane and worker nodes.

Configure kubectl access

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Install your choice of networking solutions

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml

Wait for networking pods to be ready

kubectl wait --for=condition=ready pod -l k8s-app=calico-node -n kube-system --timeout=300s

Run the control plane node join command (output of the kubeadm init) on the other master nodes.

kubeadm join 10.238.40.166:8443 --token <token> \
        --discovery-token-ca-cert-hash sha256:<hash> \
        --control-plane --certificate-key <cert-key>

Note: The certificate key is only valid for 2 hours. If it expires, generate a new one:

kubeadm init phase upload-certs --upload-certs

Verification and Health Checks

After setting up all control plane nodes, verify the cluster health:

Check all nodes are ready

kubectl get nodes -o wide

Verify control plane components

kubectl get pods -n kube-system

Check etcd cluster health

kubectl exec -n kube-system etcd-<master-node-name> -- etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  member list

Test VIP failover

# Stop keepalived on the master node that owns the VIP
sudo systemctl stop keepalived

# Verify VIP moves to another node
ip addr show | grep 10.238.40.166

# Test API access via VIP
curl -k https://10.238.40.166:8443/healthz

# Restart keepalived
sudo systemctl start keepalived

Conclusion

Congratulations! You have successfully deployed a highly available Kubernetes cluster using a stacked etcd topology with HAProxy and Keepalived. This setup provides:

Key Benefits

High Availability: Automatic failover with no single point of failure
Load Distribution: Traffic distributed across all API servers via HAProxy
Automatic Recovery: Keepalived handles VIP failover in seconds
Simplified Architecture: Stacked topology reduces complexity compared to external etcd

Cluster Capabilities

With this 3-master node configuration:

Tolerates 1 node failure while maintaining full cluster functionality
Maintains etcd quorum with 2 out of 3 members
Continues serving API requests through the remaining healthy masters
Automatically fails over VIP to operational nodes

Optional FDE in ubuntu using initrd hooks

Achyuta Das — Wed, 25 Jun 2025 06:35:33 +0000

🔍 Context:

Ubuntu’s Autoinstall (Subiquity) typically bakes full disk encryption directly into autoinstall.yaml, making it an all-or-nothing setup—either every install uses encryption, or none do. This becomes limiting when you want a single ISO image to support both encrypted and unencrypted installs without user interaction. In this blog, we will show how to use an initrd hook and a simple trigger mechanism to dynamically choose the right config at install time—enabling flexible, environment-aware deployments from a unified base image. Refer the blog post to understand more about how FDE can be done with autoinstall process.

Achyuta Das

Jun 25 '25

Full Disk Encryption (FDE) with Ubuntu Autoinstall

#linux #security #encryption

Comments

3 min read

🛠️ How it works:

The idea is to leverage Ubuntu’s initrd hooks to inject a small script that decides which autoinstall.yaml config to use — with or without full disk encryption — at runtime.

Here’s the breakdown:

Two Configs Inside the ISO
- fde/user-data → contains full disk encryption.
- nofde/user-data → no encryption. Both set of configurations are included in the ISO in a known directory.
A tiny empty image file (~350KB) will be created with a custom label "fde" will be used as the trigger. When installing, if this image is attached (eg: via USB, Floppy, cloud-init disk), it will appear in the system as: /dev/disk/by-label/fde
Initrd Hook Script A custom initrd hook script run very early in the boot process, just before we move to the installer fs. It will-
- Check if the /dev/disk/by-label/fde exists.
- If yes → copies fde/user-data to a target folder.
- If no → copies nofde/user-data instead.
Update the /boot/grub/grub.cfg to pick up the configuration for autoinstall from the target folder.

🧪 Modifications:

I used Ubuntu 24.04 live server image for this. Let's make a copy of the iso content for us to make our modifications

mkdir -p /mnt/iso ~/edit
sudo mount -o loop ubuntu-24.04.2-live-server-amd64.iso /mnt/iso
rsync -a /mnt/iso/ ~/edit
sudo umount /mnt/iso

Let's first create a folder in the base iso which will hold both fde and no-fde autoinstall configurations.

cd ~/edit
mkdir -p cidata/{fde,nofde}

cat <<EOF >> cidata/fde/user-data
... Your autoinstall.yaml content with full drive encryption enabled...
EOF
touch cidata/fde/meta-data

cat <<EOF >> cidata/nofde/user-data
... Your autoinstall.yaml content without full drive encryption enabled...
EOF
touch cidata/nofde/meta-data

We need to add a hook to the initrd. So let's first extract the existing initrd.

rm -rf ~/edit/initrd-unpacked/
mkdir -p ~/edit/initrd-unpacked
unmkinitramfs ~/edit/casper/initrd ~/edit/initrd-unpacked/

Create a file ~/edit/initrd-unpacked/main/scripts/casper-bottom/98mount-cidata with by below content

#!/bin/sh

PREREQ=""

prereqs() {
    echo "$PREREQ"
}

case $1 in
    prereqs)
        prereqs
        exit 0
        ;;
esac

mkdir -p /root/setup

if [ -b /dev/disk/by-label/fde ]; then
    cp -r /root/cdrom/cidata/fde/. /root/setup/
    echo "FDE data copied to /root/setup/" > /dev/console
else
    cp -r /root/cdrom/cidata/nofde/. /root/setup/
    echo "No FDE data found, copied nofde data to /root/setup/" > /dev/console
fi

chown -R root:root /root/setup/

exit 0

Provide executable permission to the file and Update the ORDER file located at ~/edit/initrd-unpacked/main/scripts/casper-bottom

# ORDER file before
...
/scripts/casper-bottom/61desktop_canary_tweaks "$@"
[ -e /conf/param.conf ] && . /conf/param.conf
/scripts/casper-bottom/99casperboot "$@"
[ -e /conf/param.conf ] && . /conf/param.conf

# ORDER file after
...
/scripts/casper-bottom/61desktop_canary_tweaks "$@"
[ -e /conf/param.conf ] && . /conf/param.conf
/scripts/casper-bottom/98mount-cidata "$@"
[ -e /conf/param.conf ] && . /conf/param.conf
/scripts/casper-bottom/99casperboot "$@"
[ -e /conf/param.conf ] && . /conf/param.conf

Now that the changes are done, let's re-build the initrd

cd ~/edit/initrd-unpacked
cd early
find . -print0 | cpio --null --create --format=newc > /tmp/initrd
cd ../early2
find . -print0 | cpio --null --create --format=newc >> /tmp/initrd
cd ../early3
find . -print0 | cpio --null --create --format=newc >> /tmp/initrd
cd ../main
find . | cpio --create --format=newc | xz --format=lzma >> /tmp/initrd
rm -f ~/edit/casper/initrd
mv /tmp/initrd ~/edit/casper/initrd

Update the grub.cfg for the autoinstall to pick up configuration from the /setup path on the filesystem.\

menuentry "Install Ubuntu Server" {
    set gfxpayload=keep
    linux   /casper/vmlinuz quiet autoinstall ds=nocloud\;s=/setup apparmor=0 ---  ---
    initrd  /casper/initrd
}

Generate the md5 sums.

cd ~/edit
sudo rm -f md5sum.txt
sudo bash -c "find . -type f   ! -path './isolinux/*'   ! -path './boot/*'   -print0   | xargs -0 md5sum > md5sum.txt"

Rebuild the iso.

sudo xorriso -as mkisofs -J -R -r \
      -V Ubuntu-Server \
      -o custom.iso \
      --grub2-mbr <path to your custom xx-Boot-NoEmul.img> \
      -partition_offset 16 \
      --mbr-force-bootable \
      -append_partition 2 <GUID> <path to your custom xx-Boot-NoEmul.img> \
      -appended_part_as_gpt \
      -iso_mbr_part_type <GUID>\
      -c 'boot.catalog' \
      -b 'boot/grub/i386-pc/eltorito.img' \
      -no-emul-boot -boot-load-size 4 -boot-info-table --grub2-boot-info \
      -eltorito-alt-boot \
      -e '--interval:appended_partition_2:::' \
      -no-emul-boot \
      ~/edit/

✅ Conclusion

With this setup, we have made Ubuntu's Autoinstall system a tad bit smarter and more flexible- capable of deciding at runtime whether to apply full disk encryption, all without any user interaction or multiple full ISO images; all we need is to mount an extra empty, labeled image. This approach is a bit better for reproducible, automated deployments.

Full Disk Encryption (FDE) with Ubuntu Autoinstall

Achyuta Das — Wed, 25 Jun 2025 05:29:15 +0000

🔍 Context

As system administrators and security-conscious developers, encrypting data at rest is a fundamental best practice—especially for laptops, servers in untrusted environments, or sensitive workloads. Full Disk Encryption (FDE) ensures that all data on the disk is encrypted, and can only be accessed after providing a unlocking key, thus safeguarding data even if physical access to the disk is obtained.

By leveraging cloud-init's autoinstall YAML, we can fully automate the provisioning process—including LUKS encryption, LVM setup, and embedding a drive unlocking key into the initramfs.

🛠️ Implementation Overview

Here’s a high-level overview of what we will be doing:

Use LUKS to encrypt the main data partition.
Inside the encrypted container, create a LVM volume group to manage the logical volumes.
Generate a random key file early in the installation process, which is used to unlock the encrypted volume.
Ensured this key is securely copied into the target system and embedded into the initramfs so the system can boot without manual passphrase entry.

🧪 Workflow

Early Commands: Generate a 4KB random keyfile (root.key) under /etc/cryptsetup-keys.d, with appropriate permissions.

early-commands:
    - mkdir -p /etc/cryptsetup-keys.d
    - dd if=/dev/urandom of=/etc/cryptsetup-keys.d/root.key bs=1024 count=4
    - chmod 600 /etc/cryptsetup-keys.d/root.key

Storage config: The disk was partitioned into:
- An EFI system partition (/boot/efi)
- An unencrypted /boot partition
- A third partition for encrypted data.

Create a LUKS-encrypted volume on the third partition using the key file. On top of that, create a LVM volume group (VolGroup) and allocate a root logical volume (lv_root).

  storage:
    config:
      - ptable: gpt
        match:
          size: smallest
        wipe: superblock-recursive
        preserve: false
        name: ''
        grub_device: true
        id: disk-sda
        type: disk

      - device: disk-sda
        size: 536870912  # 512MB
        wipe: superblock
        flag: boot
        number: 1
        preserve: false
        grub_device: true
        id: partition-0
        type: partition

      - fstype: fat32
        volume: partition-0
        preserve: false
        id: format-0
        type: format

      - path: /boot/efi
        device: format-0
        id: mount-0
        type: mount

      - device: disk-sda
        size: 1073741824  # 1GB
        wipe: superblock
        number: 2
        preserve: false
        grub_device: false
        id: partition-1
        type: partition

      - fstype: ext4
        volume: partition-1
        preserve: false
        id: format-1
        type: format

      - path: /boot
        device: format-1
        id: mount-1
        type: mount

      - device: disk-sda
        size: -1
        wipe: superblock
        number: 3
        preserve: false
        grub_device: false
        id: partition-2
        type: partition

      - name: cryptlvm
        volume: partition-2
        preserve: false
        keyfile: /etc/cryptsetup-keys.d/root.key
        id: dm_crypt-lvm
        type: dm_crypt

      - name: VolGroup
        devices:
          - dm_crypt-lvm
        preserve: false
        id: lvm_volgroup-0
        type: lvm_volgroup

      - name: lv_root
        volgroup: lvm_volgroup-0
        size: -1
        wipe: superblock
        preserve: false
        id: lvm_partition-root
        type: lvm_partition

      - fstype: ext4
        volume: lvm_partition-root
        preserve: false
        id: format-root
        type: format

      - path: /
        device: format-root
        id: mount-root
        type: mount

Late Commands:
- Extract the UUID of the LUKS volume and copy the key file into /etc/cryptsetup-keys.d inside the target filesystem, renaming it to match the UUID.
- Update /etc/crypttab to ensure the system knows how to unlock the encrypted volume during boot.
- Add the appropriate hook configuration for cryptsetup-initramfs and rebuilt the initramfs so the key file will be included and accessible early in the boot process.

  late-commands:
    - mkdir -p /target/etc/cryptsetup-keys.d /target/etc/cryptsetup-initramfs
    - bash -c '
        luks_uuid=$(blkid -t TYPE=crypto_LUKS -s UUID -o value);
        cp /etc/cryptsetup-keys.d/root.key "/target/etc/cryptsetup-keys.d/${luks_uuid}.key";
        chmod 600 "/target/etc/cryptsetup-keys.d/${luks_uuid}.key";
        echo "dm_crypt-lvm UUID=${luks_uuid} /etc/cryptsetup-keys.d/${luks_uuid}.key luks" > /target/etc/crypttab;
        echo "KEYFILE_PATTERN=/etc/cryptsetup-keys.d/*.key" >> /target/etc/cryptsetup-initramfs/conf-hook;
      '
    - curtin in-target --target=/target -- update-initramfs -c -k all

✅ Conclusion

With this setup, we successfully created a secure, automated Ubuntu installation with full disk encryption using LUKS and LVM. The system boots without requiring a passphrase, thanks to the securely handled key file integrated into the initramfs.
This makes the setup highly suitable for hands-off provisioning in secure environments.

Looking ahead, this foundation can be extended to further enhance security by:

Binding the LUKS key to a TPM after installation to enable secure, passphrase-free unlocks based on hardware identity.
Integrating with Secure Boot and measured boot for a trusted boot chain.

Disk Encryption using LUKS and TPM2.0

Achyuta Das — Sun, 25 May 2025 14:36:58 +0000

Introduction to LUKS

Linux Unified Key Setup (LUKS) is a disk encryption specification that encrypts block devices, such as disk drives and removable storage media. LUKS offers Full Disk Encryption (FDE) and selective partition-based encryption.

The system prompts you for a passphrase every time you boot the computer to unlock the encrypted disk. LUKS encrypted volumes can be automatically unlocked (without the need to provide a passphrase at boot), using a Trusted Platform Module 2.0 (TPM 2.0) policy

Introduction to TPM2.0

The Trusted Platform Module 2.0 (TPM) is a hardware-based system security feature that securely stores passwords, certificates, and encryption keys to authenticate the platform. It is embedded in the server motherboard.

TPM supports auto unlocking of the encrypted disk during system startup without requiring any user intervention.

Setup Information

This was performed on vanilla Ubuntu 24.04 Partition based encryption. Ubuntu was installed on a VM which was created using multipass on top of Hyper-V, with TPM and secure boot enabled for the VM.

The steps should work on a bare-metal node as well.

root@honeyglaze:~# uname -r
6.8.0-60-generic

root@honeyglaze:~# cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.2 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

# Validating TPM device
root@honeyglaze:~# ls -l /dev/tpm*
crw-rw---- 1 tss root  10,   224 May 22 05:49 /dev/tpm0
crw-rw---- 1 tss tss  253, 65536 May 22 05:49 /dev/tpmrm0

# Validating secure boot
root@honeyglaze:~# mokutil --sb-state
SecureBoot enabled

# cryptsetup version
root@honeyglaze:~# cryptsetup --version
cryptsetup 2.7.0 flags: UDEV BLKID KEYRING FIPS KERNEL_CAPI HW_OPAL

root@honeyglaze:~# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda       8:0    0   10G  0 disk                    #### We will be working on /dev/sda disk
sdb       8:16   0   20G  0 disk
├─sdb1    8:17   0   19G  0 part /
├─sdb14   8:30   0    4M  0 part
├─sdb15   8:31   0  106M  0 part /boot/efi
└─sdb16 259:0    0  913M  0 part /boot
sr0      11:0    1   54K  0 rom

Encrypting the disk

Use the cryptsetup luksFormat command to encrypt the disk. Provide a passphrase for encrypting the partition.

root@honeyglaze:~# cryptsetup luksFormat /dev/sda

WARNING!
========
This will overwrite data on /dev/sda irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for /dev/sda:
Verify passphrase:

Create a logical device-mapper that can be mounted to the encrypted partition. Provide the same passphrase given in the previous step to unlock the encrypted drive.

root@honeyglaze:~# cryptsetup luksOpen /dev/sda cryptpart
Enter passphrase for /dev/sda:

Format the new partition with the ext4 file system.

root@honeyglaze:~# mkfs.ext4 /dev/mapper/cryptpart
mke2fs 1.47.0 (5-Feb-2023)
Creating filesystem with 2617344 4k blocks and 655360 inodes
Filesystem UUID: 73c76f23-804d-431d-84be-4767a6dc81c7
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information:  done

Mount the new file system.

root@honeyglaze:~# mkdir -p /mnt/cryptpart

root@honeyglaze:~# mount /dev/mapper/cryptpart /mnt/cryptpart/

root@honeyglaze:~# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
sda           8:0    0   10G  0 disk
└─cryptpart 252:0    0   10G  0 crypt /mnt/cryptpart
sdb           8:16   0   20G  0 disk
├─sdb1        8:17   0   19G  0 part  /
├─sdb14       8:30   0    4M  0 part
├─sdb15       8:31   0  106M  0 part  /boot/efi
└─sdb16     259:0    0  913M  0 part  /boot
sr0          11:0    1   54K  0 rom

Adding a recovery key.

# Generate a key
root@honeyglaze:~# openssl rand -base64 32 | cut -c1-32 | tee keyfile.txt
UXItVWYLicvja4u4PdYtMZ/iPwcufRGz

# Use luksAddKey to add the generate key, enter the passphrase (given in step 1) when prompted.
root@honeyglaze:~# cryptsetup luksAddKey /dev/sda keyfile.txt
Enter any existing passphrase:

Validate the key slots.

# Key slot 0 and 1 are for the primary passphrase and the recovery key

root@honeyglaze:~# cryptsetup luksDump /dev/sda
LUKS header information
Version:        2
Epoch:          4
Metadata area:  16384 [bytes]
Keyslots area:  16744448 [bytes]
UUID:           608d9da9-a3fa-4a4e-8923-c3e4d95247a3
Label:          (no label)
Subsystem:      (no subsystem)
Flags:          (no flags)

Data segments:
  0: crypt
        offset: 16777216 [bytes]
        length: (whole device)
        cipher: aes-xts-plain64
        sector: 4096 [bytes]

Keyslots:
  0: luks2
        Key:        512 bits
        Priority:   normal
        Cipher:     aes-xts-plain64
        Cipher key: 512 bits
        PBKDF:      argon2id
        Time cost:  6
        Memory:     1048576
        Threads:    2
        Salt:       c4 ec 58 59 3a 34 48 bb 01 44 94 6f 7f 41 4d d9
                    a8 74 ba a6 1a 57 60 86 f1 19 69 e5 c2 d6 e7 30
        AF stripes: 4000
        AF hash:    sha256
        Area offset:32768 [bytes]
        Area length:258048 [bytes]
        Digest ID:  0
  1: luks2
        Key:        512 bits
        Priority:   normal
        Cipher:     aes-xts-plain64
        Cipher key: 512 bits
        PBKDF:      argon2id
        Time cost:  5
        Memory:     1048576
        Threads:    2
        Salt:       3e fb 84 89 4a c2 53 bb 4e d8 0f d1 aa a3 f1 81
                    e1 31 8a af 67 69 b8 96 c1 da ba a1 d7 db 2f 72
        AF stripes: 4000
        AF hash:    sha256
        Area offset:290816 [bytes]
        Area length:258048 [bytes]
        Digest ID:  0
Tokens:
Digests:
  0: pbkdf2
        Hash:       sha256
        Iterations: 246375
        Salt:       2a 25 1d 0e b9 33 2b 8b b6 8f 28 fc 71 4e 51 08
                    78 ba d8 36 45 21 13 b4 6e 66 0a a5 85 f1 a9 eb
        Digest:     2a d1 7b 8a e2 b2 3d f1 17 29 7a 16 54 1e 3a 67
                    a8 f8 f0 f7 79 c1 a3 06 2b bd bf 09 9d 3b 36 4d

Decrypt a LUKS volume with Clevis and TPM

Clevis is a pluggable framework for automated decryption. It can be used to provide automated decryption of data or even automated unlocking of LUKS volumes. I went with clevis since in case of passphrase in TPM chip is corrupted/unreachable, clevis automatically prompts for passphrase/recovery key without requiring intervention to update /etc/crypttab to drop TPM device.

Install the required packages.

apt-get install dracut-core clevis clevis-tpm2 clevis-luks clevis-dracut tss2

Bind LUKS encryption key/password to TPM2.0. Enter the passphrase when prompted.

# "-s 2" specifies the key slot. 0 and 1 slots are already in use.
root@honeyglaze:~# clevis luks bind -d /dev/sdb tpm2 '{"pcr_bank":"sha256","pcr_ids":"7"}'
Enter existing LUKS password:

Validate the binding.

root@honeyglaze:~# clevis luks list -d /dev/sda
2: tpm2 '{"hash":"sha256","key":"ecc","pcr_bank":"sha256","pcr_ids":"7"}'

root@honeyglaze:~# systemd-cryptenroll /dev/sda
SLOT TYPE
   0 password   # passphrase
   1 password   # recovery key
   2 other      # clevis binding

Let's try to unlock the drive and see whether we are getting the prompt to enter the password or not.

# First we need to unmount the disk
root@honeyglaze:~# umount /mnt/cryptpart

root@honeyglaze:~# cryptsetup luksClose cryptpart

root@honeyglaze:~# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda       8:0    0   10G  0 disk
sdb       8:16   0   20G  0 disk
├─sdb1    8:17   0   19G  0 part /
├─sdb14   8:30   0    4M  0 part
├─sdb15   8:31   0  106M  0 part /boot/efi
└─sdb16 259:0    0  913M  0 part /boot
sr0      11:0    1   54K  0 rom

# unlock command should not give a prompt to enter the passphrase
root@honeyglaze:~# clevis luks unlock -d /dev/sda -n cryptpart

root@honeyglaze:~# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
sda           8:0    0   10G  0 disk
└─cryptpart 252:0    0   10G  0 crypt
sdb           8:16   0   20G  0 disk
├─sdb1        8:17   0   19G  0 part  /
├─sdb14       8:30   0    4M  0 part
├─sdb15       8:31   0  106M  0 part  /boot/efi
└─sdb16     259:0    0  913M  0 part  /boot
sr0          11:0    1   54K  0 rom

Enable automatic decryption during boot

Add an entry to /etc/crypttab for decrypting the volume at boot time.

# Use blkid to get the UUID of the encrypted drive
root@honeyglaze:~# blkid -t TYPE=crypto_LUKS
/dev/sda: UUID="608d9da9-a3fa-4a4e-8923-c3e4d95247a3" TYPE="crypto_LUKS"

root@honeyglaze:~# cat /etc/crypttab
# <target name> <source device>         <key file>      <options>
cryptpart    UUID=608d9da9-a3fa-4a4e-8923-c3e4d95247a3 none luks,discard,nofail

Rebuild the initramfs and reboot.

root@honeyglaze:~# dracut -f

......
dracut[I]: *** Creating image file '/boot/initrd.img-6.8.0-60-generic' ***
dracut[I]: Using auto-determined compression method 'pigz'
dracut[I]: *** Creating initramfs image file '/boot/initrd.img-6.8.0-60-generic' done ***

root@honeyglaze:~# reboot

Validate boot logs.


root@honeyglaze:~# journalctl -b > boot.log

root@honeyglaze:~# cat boot.log | grep tpm
May 22 08:10:24 localhost kernel: tpm_crb VTPM0101:00: [Firmware Bug]: Bad ACPI memory layout
May 22 08:10:24 localhost kernel: tpm_crb VTPM0101:00: [Firmware Bug]: Bad ACPI memory layout
May 22 08:10:26 honeyglaze systemd[1]: systemd-tpm2-setup-early.service - TPM2 SRK Setup (Early) was skipped because of an unmet condition check (ConditionSecurity=measured-uki).
May 22 08:10:26 honeyglaze systemd[1]: systemd-tpm2-setup.service - TPM2 SRK Setup was skipped because of an unmet condition check (ConditionSecurity=measured-uki).
May 22 08:10:29 honeyglaze systemd[1]: systemd-tpm2-setup-early.service - TPM2 SRK Setup (Early) was skipped because of an unmet condition check (ConditionSecurity=measured-uki).
May 22 08:10:29 honeyglaze systemd[1]: systemd-tpm2-setup.service - TPM2 SRK Setup was skipped because of an unmet condition check (ConditionSecurity=measured-uki).
May 22 08:10:32 honeyglaze systemd[1]: tpm-udev.path - Handle dynamically added tpm devices was skipped because of an unmet condition check (ConditionVirtualization=container).

root@honeyglaze:~# cat boot.log | grep clevis
May 22 08:10:24 localhost systemd[1]: Started clevis-luks-askpass.path - Forward Password Requests to Clevis Directory Watch.
May 22 08:10:26 localhost systemd[1]: clevis-luks-askpass.path: Deactivated successfully.
May 22 08:10:26 localhost systemd[1]: Stopped clevis-luks-askpass.path - Forward Password Requests to Clevis Directory Watch.
May 22 08:10:24 localhost systemd[1]: Started clevis-luks-askpass.path - Forward Password Requests to Clevis Directory Watch.
May 22 08:10:26 localhost systemd[1]: clevis-luks-askpass.path: Deactivated successfully.
May 22 08:10:26 localhost systemd[1]: Stopped clevis-luks-askpass.path - Forward Password Requests to Clevis Directory Watch.
May 22 08:10:26 honeyglaze systemd[1]: Started clevis-luks-askpass.path - Forward Password Requests to Clevis Directory Watch.
May 22 08:10:29 honeyglaze systemd[1]: Started clevis-luks-askpass.service - Forward Password Requests to Clevis.
May 22 08:10:29 honeyglaze clevis-luks-askpass[809]: Unlocked /dev/disk/by-uuid/608d9da9-a3fa-4a4e-8923-c3e4d95247a3 (UUID=608d9da9-a3fa-4a4e-8923-c3e4d95247a3) successfully
May 22 08:10:36 honeyglaze systemd[1]: clevis-luks-askpass.service: Deactivated successfully.
May 22 08:10:36 honeyglaze systemd[1]: clevis-luks-askpass.service: Consumed 1.283s CPU time.

Bonus Info

TPM is a hardware chip that can securely store keys and perform cryptographic operations.
pcr_id refers to the Platform Configuration Register (PCR) index that is used by the TPM (Trusted Platform Module) to store measurements of system state during the boot process. These registers in TPM (numbered 0–23 for TPM 2.0) that store hashes of system components during boot. These values change if the boot environment changes, which makes them great for detecting tampering.
When you bind a LUK S volume to TPM2 using Clevis , you specify one or more pcr_ids —for example, with clevis luks bind -d /dev/sdX tpm2 '{"pcr_ids":"0,7"}'. This command seals the decryption key to the current values of PCRs 0 and 7, which represent specific measurements of the system state (like firmware, bootloader, and kernel). If these PCR values change—due to modifications in boot parameters, kernel, initramfs, or the bootloader—the TPM will not unseal the key, and the system will prompt for the LUKS passphrase. This mechanism ensures that the disk can only be automatically decrypted in a known, trusted, and unmodified boot environment.
Execute sudo tpm2_pcrread sha256:0,1,2,3,4,5,6,7 to display the hashes. Reboot the validate whether these hash values are changing.

Introducing inmem – Lightweight Go Cache Engine with Built-in Sharding, Transaction, and Eviction

Achyuta Das — Wed, 23 Apr 2025 09:16:12 +0000

After spending way too much time, I went with a totally original, never-before-seen name for an in-memory cache: inmem. Creative, right?
Jokes aside, it all started as a side experiment and went through more refactor cycles than I care to admit. But now, it’s grown into something clean, fast, and actually useful. Introducing inmem— an embedded caching library written in pure Go, designed with simplicity and performance in mind. It supports eviction policies, sharding for concurrency, and transactions for atomic operations.

So, what is inmem?

It is a fast, embedded in-memory caching library written in Go. It's like a mini key-value store, living happily inside your app, keeping your data hot and your latency low.

💡 Why I built it?

I was bored

⚙️ Key Features

Eviction Support: Keep memory usage in check with support of an optimized TTL-based eviction. It also supports an efficient LRU and LFU. ARC (Adaptive Replacement Cache) is next in line!
Sharding: Out of the box sharding support for not overloading a single map.
Transactions: Atomic read/write/delete operations across multiple keys. No need to reinvent locking or worry about inconsistent state.
Thread-safe: Go nuts with goroutines — it’s internal locking is shard-aware and handles the dirty work for you.
Standalone eviction cache: Use the eviction cache (LRU or LFU) independently.
Persistence: Periodically save cache data to disk and load it on startup.

🚀 Quick Start

Install is the usual way:

go get github.com/achu-1612/inmem

🧪 Usage

Basic Usage

A simple example showing how to store and retrieve a custom struct (User) from the cache. Don't forget to gob.Register your types if you're storing structs!

package main

import (
    "context"
    "fmt"
    "github.com/achu-1612/inmem"
)

type User struct {
    Name string
    Age  int
}

func main() {
    gob.Register(User{})
    cache, err := inmem.New(context.Background(), inmem.Options{})
    if err != nil {
        panic(err)
    }

    cache.Set("key", User{Name: "Achu", Age: 25}, 0)
    value, found := cache.Get("key")
    if found {
        fmt.Println("Found value:", value)
    } else {
        fmt.Println("Value not found")
    }
}

Sharded Cache

Enable sharding to reduce lock contention and improve performance under concurrent access. You can configure the number of shards as needed.

    cache, err := inmem.New(context.Background(), inmem.Options{
        Sharding:   true,
        ShardCount: 4,
    })

Transactions

Enable transactional support to perform atomic operations (like multi-key updates/deletes) safely. Currently supports optimistic and atomic transactions.

    cache, err := inmem.New(context.Background(), inmem.Options{
        TransactionType: inmem.TransactionTypeOptimistic,
    })

Persistence

Enable persistence to periodically write cache data to disk. Useful if you want the cache to survive restarts.

    cache, err := inmem.New(context.Background(), inmem.Options{
        Sync:           true,
        SyncInterval:   time.Minute,
        SyncFolderPath: "cache_data",
    })

Eviction

Configure the cache to automatically evict entries based on policy (e.g., LRU) when the size limit is reached.

    cache, err := inmem.New(context.Background(), inmem.Options{
        EvictionPolicy: eviction.PolicyLRU,,
        MaxSize: 2,
    })
    if err != nil {
        panic(err)
    }

    cache.Set("key1", "value", 0)
    cache.Set("key2", "value", 0)
    cache.Get("key2")
    cache.Set("key3", "value", 0) // key1 will be evicted as key2 has access frequency as 2 and key1 has 1

Standalone Eviction cache

You can also use the underlying eviction engine independently if all you need is a simple key-value store with LRU or LFU eviction.

    cache := eviction.New(eviction.Options{
        Capacity: 1000,
        Policy:   eviction.PolicyLRU,
        // Policy:  eviction.PolicyLFU,
    })

🎯 Final Thoughts

It all started as a simple experiment, but it’s grown into something I'm genuinely excited to share. Whether you're building a blazing-fast API, a CLI tool, or just need a clean way to manage in-memory data — I hope inmem makes your life a bit easier (and faster).

If you give it a try and something breaks — that’s probably my fault.
If you try it and it works flawlessly — tell everyone it was intentional.

The project is open-source, and I’d love your feedback, suggestions, or contributions. Star it, fork it, break it, fix it — all welcome!

🙌 Thanks for Reading!

If you made it this far, you either really like caching or you're just incredibly patient. Either way, thanks for stopping by!

Have a good day.

Fluent-bit as a sidecar in Pod

Achyuta Das — Mon, 20 Dec 2021 14:03:10 +0000

Most modern applications have some kind of logging mechanism. Likewise, container engines are designed to support logging. The easiest and most adopted logging method for containerized applications is writing to standard output and standard error streams. However, the native functionality provided by a container engine or runtime is usually not enough for a complete logging solution.

Logging architectures require a separate backend to store, analyze, and query logs. Kubernetes does not provide a native storage solution for log data. Instead, many logging solutions integrate with Kubernetes.

In this article, the goal is to collect standard output logs and ship them to a centralized store. The log collector can be configured as a sidecar in the pod or as a daemonset in the cluster.

The Architecture:

Fluent Bit is used to collect and ship the standard output logs from the pod. It is an open-source Log Processor and Shipper, designed with performance in mind: high throughput with low CPU and Memory usage. Influx DB, an open-source time-series database, is used as the centralized store for the logs.

The setup:

Wait! How to access the stdout logs from a pod?

The standard output logs are written to /var/log/pods/{NAMESPACE}_{POD_NAME}_{POD_ID}/{CONTAINER_NAME}/*.log files . We just need to tail all the log files present in the above location.

Fluent Bit Configuration

A configmap is created to hold the fluent bit config template.

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config-template
data:
  fluent-bit-template.conf: |
    [INPUT]
      Name tail
      Path /var/log/pods/${NAMESPACE}_${POD_NAME}_${POD_ID}/${CONTAINER_NAME}/*.log
    [OUTPUT]
      Name          influxdb
      Match         *
      Host          ${INFLUX_HOST}
      Port          ${INFLUX_PORT}
      Database      ${INFLUX_DB}
      Sequence_Tag  _seq
      http_token    ${INFLUX_TOKEN}
      Bucket        ${INFLUX_BUCKET}
      Org           ${INFLUX_ORG}

Fluent Bit connection to InfluxDB

A secret object is created containing access configuration and credentials for InfluxDB.

apiVersion: v1
kind: Secret
metadata:
  name: influx-db-cred
data:
  INFLUX_HOST: <<TO_BE_UPDATED>>
  INFLUX_PORT: <<TO_BE_UPDATED>>
  INFLUX_DB: <<TO_BE_UPDATED>>
  INFLUX_TOKEN: <<TO_BE_UPDATED>>
  INFLUX_BUCKET: <<TO_BE_UPDATED>>
  INFLUX_ORG: <<TO_BE_UPDATED>>

Pod configuration for running Fluent Bit as a sidecar

Volumes are used to hold the fluent-bit config file template, the actual template file. Pod logs present in the host are mounted to the sidecar.

  volumes:
  - name: fluent-bit-config-template
    configMap:
      name: fluent-bit-config-template
      defaultMode: 0777
  - name: app-log
    hostPath:
      path: /var/log/pods
      type: Directory
  - name: fluent-bit-config
    emptyDir: {}

A valid fluent-bit config file is generated from the defined template. All the required details and credentials are injected into the init-container, which will prepare the fluent-bit config file from the template. Using K8s Downward API, Pod information are exposed.

  initContainers:
  - name: fluent-bit-config-manager
    image: busybox:latest
    volumeMounts:
      - mountPath: /fluent-bit-template.conf
        name: fluent-bit-config-template
        subPath: fluent-bit-template.conf
      - mountPath: /conf
        name: fluent-bit-config
    envFrom:
    - secretRef:
        name: influx-db-cred
    env:
    - name: POD_ID
      valueFrom:
        fieldRef:
          fieldPath: metadata.uid
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace
    - name: CONTAINER_NAME
      value: *container1
    command: ["/bin/sh"]
    args:
    - -c
    - eval "echo \"$(cat /fluent-bit-template.conf)\"" > /conf/fluent-bit.conf

The fluent-bit sidecar container requires the access to the log files created by pods and the generated configuration file. Both of which are mounted to the sidecar using volumes.

  - name: fluent
    image: fluent/fluent-bit:1.8
    volumeMounts:
      - mountPath: /fluent-bit/etc/fluent-bit.conf
        name: fluent-bit-config
        subPath: fluent-bit.conf
      - mountPath: /var/log/pods
        name: app-log

Visualizing the logs in InfluxDB dashboard

Running a query on the configured bucket will return a result set containing all the logs shipped by the sidecar.

What's next?

Explore different input and output plugins supported by Fluent Bit.
Use parser and filter to format the logs
Build your own fluent-bit plugin
Run Fluent Bit as a deamonset in the cluster.

Here are all the manifests used for the setup.

CI/CD for Kubernetes using GitHub Actions, and Keel

Achyuta Das — Tue, 13 Apr 2021 12:38:21 +0000

In this article, the goal is to show how to set up a containerized application in Kubernetes with a very simple CI/CD pipeline to manage deployment using GitHub Actions and Keel.

Before we start:

Kubernetes, also known as K8s, is an open-source container orchestration system for automating deployment, scaling, and management.

Keel is a K8s operator to automate Helm, DaemonSet, Stateful & Deployment updates. It’s open-source, self-hosted with zero requirements of CLI/API, and comes with a beautiful and insightful dashboard.

GitHub Actions makes it easy to automate all your software workflows, now with world-class CI/CD. Build, test, and deploy your code right from GitHub.

Workflow:

There are basically two steps in the workflow.

You will push some changes to the GitHub repo. A workflow will be triggered, which will build the docker image of our application and push the image to the Docker registry.
Keel will get notified of the updated image. Based on the update policy, the deployment will be updated in the configured cluster.

Step 1:

In the first step, we will prepare the GitHub repo to trigger workflows.

The repo, aka-achu/go-kube contains a simple web application written in golang and a Dockerfile, which will be used to build a docker image of the application. You can maintain any number of environments for your application like Production, QnA, Staging, Development, etc. For sake of simplicity, we will be maintaining only two deployment environments of the application.
There are only two branches in the repo.

main branch (for Production environment)

name: Stable Build
on:
  push:
    tags:
      - "*.*.*"
...
      - name: Set tag in env
        run: echo "TAG=${GITHUB_REF#refs/*/}" >> $GITHUB_ENV
...
          tags: runq/go-kube:${{ env.TAG }}, runq/go-kube:latest

The stable workflow will be triggered when a tag is pushed to GitHub. In the workflow, the tag associated with the commit will be used as the docker image tag.

dev branch (for Development/Staging environment)

name: Development Build
on:
  push:
    branches: [ dev ]
...
      - name: Set short commit hash in env
        run: echo "COMMIT_SHA=$(echo $GITHUB_SHA | cut -c1-7)" >> $GITHUB_ENV
...
          tags: runq/go-kube:dev-${{ env.COMMIT_SHA }}

The dev workflow will be triggered when any changes are pushed to the dev branch. For development builds, instead of git tags, we will use the short commit hash of length 7, which we often see in GitHub. The docker image tag of the development build image will be dev-SHORT_COMMIT_SHA.

Both workflows aim to integrate the code, maybe run some tests, build the docker image, and update the image registry. Till this point, we have completed Continuous Integration and Continuous Delivery.

Step 2:

In this step, we will automate our deployment update. We will be using the K8s LoadBalancer service in our workflow. So, if you're using an on-premise cluster then you can use MetalLB, which is a load-balancer implementation for bare metal Kubernetes clusters.

Install keel:

Keel doesn't need a database. Keel doesn't need any persistent disk. It gets all required information from your cluster

kubectl apply -f https://sunstone.dev/keel?namespace=keel&username=admin&password=admin&tag=latest

The above command will deploy Keel to keel namespace with basic authentication enabled and admin dashboard. You can provide an admin password while applying the manifest or you can download the manifest and replace the default password.

            # Basic auth (to enable UI/API)
            - name: BASIC_AUTH_USER
              value: admin
            - name: BASIC_AUTH_PASSWORD
              value: admin
            - name: AUTHENTICATED_WEBHOOKS
              value: "false"

Keel policies:

In keel, we use policies to define when we want our application/deployment to get updated. Following semver best practices allows you to safely automate application updates. Keel supports many different policies to update resources. For now, we will use only all and glob policies to update our deployment.

all: update whenever there is a version bump (1.0.0 -> 1.0.1) or a new prerelease created (ie: 1.0.0 -> 1.0.1-rc1)
glob: use wildcards to match versions (eg: dev-* in our scenario)

How Keel will get notified:

Webhooks- when a new image is pushed to the registry, keel will get notified by the added webhook in DockerHub, and based on the configured update policy, the deployment will be updated.
Polling- when an image with a non-semver style tag is used (ie: latest) Keel will monitor SHA digest. If a tag is semver - it will track and notify providers when new versions are available.

Configuring webhook:

First, we need to get the External IP address of the keel service.

kubectl get all -n keel

The output will something like this:

Now, that we have the external address of the service/keel, we will add a webhook for our repository in DockerHub. The URL for the hook will be http://<External-IP>:9300/v1/webhooks/dockerhub. Now pushing a new docker image will trigger an HTTP call-back.

If you don't want to expose your Keel service - the recommended solution is webhookrelay which can deliver webhooks to your internal Keel service through a sidecar container. Here is how you can set up the sidecar.

Configuring deployment manifest:

To configure our staging deployment, which runs an image having a tag of dev-SHORT_COMMIT_SHA, we will use the glob policy. We will specify our policy/update rule using annotations under the metadata of the deployment manifest. Like-

...
  annotations:
    keel.sh/policy: "glob:dev-*"
...

To configure our production deployment, which runs an image having a semver tag, we will use the all policy. This will update our production deployment when it encounters any version bump in the tag. We will specify our policy/update rule using annotations under the metadata of the deployment manifest. Like-

...
  annotations:
    keel.sh/policy: all
...

Testing the workflow:

Now, that we are done with the setup, a push to the dev branch, will update the staging deployment and a tagged release will update the production deployment.

Push changes to the dev branch
Development workflow will be triggered which will build a docker image
A Docker image with the tag dev-SHORT_COMMIT_SHA will be pushed to the DockerHub registry.
Keel will get notified by DockerHub using the webhook
Keel will validate, whether the new image tag satisfies the policy we have specified (all tags starting with dev- will qualify for the update process. eg: dev-c722d00, dev-0ca740e)
If the tag qualifies for the update, keel will create a new replicaset with the new image. Once the pods are ready, keel will scale the number of replicas in the old replicaset to 0.

The release of a tag will trigger a similar flow of events.

Visualizing using Keel dashboard:

You can access the dashboard at External-IP:9300 or by using the NodePort. Use the same credentials which you had set while setting up the keel.
You can -

View resources managed by keel
Get an audit of the changes done by keel
Change/Pause update policies of resources
Approve updates

What's next?

Check out the keel documentation to explore more features.

Enabling approvals for udpates
Setting up notification pipelines
Supported webhook triggers and polling
Use helm templating for update
Updating DaemonSets, StatefulSets, etc.

Here are all the workflow and deployment manifests used.