DEV Community: giveitatry

Deploying RustFS (Single Node) on k3s

giveitatry — Fri, 24 Apr 2026 08:08:04 +0000

RustFS is an S3-compatible object storage system designed for high performance and distributed environments. For development, testing, or edge deployments, it can be run in standalone mode (single node, single disk).

This guide covers:

Helm-based installation
Correct credential configuration
Optional ingress exposure
Verification steps
Troubleshooting common issues

1. Prerequisites

Ensure the following:

Running k3s cluster
kubectl configured
Helm installed

Check cluster:

kubectl get nodes

2. Add RustFS Helm repository

helm repo add rustfs https://charts.rustfs.com
helm repo update

3. Create namespace

kubectl create namespace rustfs

4. Prepare values.yaml

This is the recommended configuration for single-node deployment.

mode:
  standalone:
    enabled: true
  distributed:
    enabled: false

replicaCount: 1

image:
  repository: rustfs/rustfs
  tag: latest

storageclass:
  name: local-path

secret:
  rustfs:
    access_key: admin
    secret_key: supersecretpassword

ingress:
  enabled: false

tls:
  enabled: false

5. Install RustFS

helm install rustfs rustfs/rustfs \
  -n rustfs \
  -f values.yaml

6. Verify deployment

Check pods:

kubectl get pods -n rustfs

Expected:

rustfs-0   Running

7. Access RustFS

Option A: Port-forward (recommended)

kubectl port-forward svc/rustfs-svc 9000:9000 -n rustfs

Then:

API: http://localhost:9000
Console: http://localhost:9001

Option B: Enable ingress (optional)

Update values.yaml:

ingress:
  enabled: true
  className: traefik
  hosts:
    - host: rustfs.127.0.0.1.nip.io
      paths:
        - path: /
          pathType: Prefix

Apply:

helm upgrade rustfs rustfs/rustfs -n rustfs -f values.yaml

8. Login

Default credentials (as configured):

Username: admin
Password: supersecretpassword

Access the web console on port 9001.

9. Key Configuration Notes

Standalone vs Distributed

standalone = single pod, one volume
distributed = multiple pods, erasure coding

Do not mix both modes.

Credentials

RustFS Helm chart uses:

secret.rustfs.access_key
secret.rustfs.secret_key

Not:

auth.*

Storage

Default:

storageclass.name: local-path

Suitable for k3s environments.

10. Troubleshooting

1. Login does not work

Cause:

Wrong Helm keys used (auth.* instead of secret.rustfs.*)
Secret not updated

Fix:

kubectl get secret -n rustfs
kubectl delete secret <secret-name> -n rustfs
helm upgrade rustfs rustfs/rustfs -n rustfs -f values.yaml

2. Pods fail with "not first disk" or quorum errors

Cause:

Distributed mode enabled unintentionally
Invalid cluster topology

Fix:

Ensure:

mode:
  standalone:
    enabled: true
  distributed:
    enabled: false

3. TCP connection errors between nodes

Cause:

Port mismatch (9000 vs 9001)
Incorrect service configuration

Fix:

Ensure consistent port usage
Verify endpoints:

kubectl get endpoints -n rustfs

4. Ingress fails with validation error

Error:

spec.rules[0].http.paths: Required value

Cause:

Missing paths definition

Fix:

paths:
  - path: /
    pathType: Prefix

5. Ingress not reachable

Cause:

Missing ingress controller
Wrong class name

Fix:

For k3s:

ingress:
  className: traefik

6. Cannot access service externally

Cause:

Using ClusterIP without port-forward

Fix:

kubectl port-forward svc/rustfs-svc 9000:9000 -n rustfs

7. Credentials do not change after upgrade

Cause:

Kubernetes Secret not overwritten

Fix:

kubectl delete secret <secret-name> -n rustfs
helm upgrade ...

Conclusion

Running RustFS in standalone mode on k3s is straightforward when:

Helm values are correctly defined
Secrets use the correct keys
Distributed mode is disabled

This setup provides a lightweight, S3-compatible storage backend suitable for development and edge use cases, while preserving a clear upgrade path to distributed deployments.

Backup and Restore of a K3s Cluster Using Velero

giveitatry — Fri, 24 Apr 2026 07:01:18 +0000

This guide provides a practical, production-oriented walkthrough of installing, configuring, and troubleshooting Velero for backing up a K3s cluster. It focuses on reliability and reproducibility rather than theory.

1. Overview

Velero is an open-source tool designed to:

Back up Kubernetes resources (Deployments, Services, CRDs, etc.)
Optionally back up persistent volumes
Restore workloads to the same or a different cluster

In K3s environments, Velero is commonly used for:

Disaster recovery
Cluster migration
CI/CD environment replication

2. Installing Velero CLI on Linux

2.1 Download

You already identified the correct approach. Use:

curl -L https://github.com/velero-io/velero/releases/download/v1.18.0/velero-v1.18.0-linux-amd64.tar.gz | tar xz

This downloads and extracts:

velero-v1.18.0-linux-amd64/

2.2 Install binary system-wide

cd velero-v1.18.0-linux-amd64
sudo mv velero /usr/local/bin/

2.3 Verify installation

velero version

Expected output:

Client:
  Version: v1.18.0

At this stage, the server part is not yet installed — that comes next.

3. Preparing Storage Backend

Velero requires object storage (S3-compatible).

Supported options:

AWS S3
MinIO (recommended for on-prem)
Hetzner Object Storage
Azure Blob / GCP

3.1 Create credentials file

Example:

cat <<EOF > credentials-velero
[default]
aws_access_key_id=<ACCESS_KEY>
aws_secret_access_key=<SECRET_KEY>
EOF

4. Installing Velero in K3s

Velero runs inside the cluster as a controller.

4.1 Installation command

Example (S3-compatible storage):

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.8.0 \
  --bucket velero \
  --secret-file ./credentials-velero \
  --use-volume-snapshots=false \
  --backup-location-config \
    region=hel1,s3ForcePathStyle="true",s3Url=https://<OBJECT_STORAGE_ENDPOINT>:<PORT-IF-NEEDED>

Key parameters explained

--provider aws
Required even for S3-compatible systems
--plugins
Enables S3 integration
--bucket
Must already exist in storage
region=minio
Required (even if not AWS) - any region will work
s3ForcePathStyle="true"
Required for most non-AWS providers
s3Url
Must be reachable from inside the cluster

4.2 Verify installation

kubectl get pods -n velero

Expected:

velero-xxxxx   Running

4.3 Validate storage

velero backup-location get

Expected:

PHASE: Available

This is the most important health indicator.

5. Creating Backups

5.1 Full cluster backup

velero backup create full-backup

5.2 Check status

velero backup get

Expected:

STATUS: Completed

5.3 Detailed inspection

velero backup describe full-backup --details

6. Restoring from Backup

velero restore create --from-backup full-backup

This will:

recreate resources
restore metadata
optionally restore volumes (if configured)

7. Automating Backups

Example: daily backup at 02:00

velero schedule create daily-backup \
  --schedule="0 2 * * *"

8. Persistent Volume Backups (Important)

By default (as in your setup):

--use-volume-snapshots=false

This means:

Kubernetes objects are backed up
Persistent data is NOT

Options:

Option A — CSI snapshots (preferred)

Requires storage support

Option B — Node-agent (Restic)

--use-node-agent

Without this, your backups are incomplete for stateful workloads.

9. Troubleshooting

9.1 CLI works but server not found

Error:

no matches for velero.io/v1

Cause:

Velero not installed in cluster

Fix:

run velero install

9.2 Backup fails with `FailedValidation`

Common causes:

missing region
invalid storage config
bucket not reachable

Fix:
Ensure:

region: minio

9.3 MissingRegion error

MissingRegion: could not find region configuration

Fix:
Add:

region=minio

9.4 Storage unavailable

BackupStorageLocation is in unavailable state

Check:

velero backup-location get

If not Available, verify:

endpoint URL (must be reachable from cluster)
credentials
bucket existence

9.5 Connection timeout

dial tcp ... i/o timeout

Cause:

cluster cannot reach storage

Fix:

verify network access from pod:

kubectl run test --rm -it --image=busybox -- sh
wget -O- https://<endpoint>

9.6 Incorrect protocol (HTTP vs HTTPS)

Many providers require HTTPS.

Incorrect:

http://...

Correct:

https://...

9.7 Plugin process exited (not an error)

plugin process exited

This is normal behavior:

plugin runs per request
exits after execution

No action required.

9.8 Inconsistent logs (0/0/1 state)

available/unavailable/unknown: 0/0/1

Occurs during startup.

Ignore if:

velero backup-location get

shows:

Available

10. Recovery Strategy for K3s

To restore a cluster on a new server:

Install K3s
Install Velero
Configure same storage
Run:

velero restore create --from-backup <backup-name>

This reconstructs:

workloads
configs
namespaces

11. Best Practices

Always verify backups with test restores
Use external object storage (not same node)
Automate backups (schedules)
Enable volume backup for production workloads
Version control your Velero installation config

Restarting a Kubernetes Deployment from GitLab CI

giveitatry — Thu, 23 Apr 2026 10:03:36 +0000

Introduction

In many delivery workflows, a full redeploy is unnecessary. A rollout restart is sufficient to force Pods to be recreated (for example after updating ConfigMaps, Secrets, or re-pulling an image with the same tag). This article describes how to trigger:

kubectl rollout restart deployment/<name>

from GitLab CI against Kubernetes (including k3s), using a minimal and secure setup.

Kubernetes-side requirements

To allow a CI job to operate on the cluster, you need:

ServiceAccount – identity used by the CI job
Role (RBAC) – permissions (minimal scope)
RoleBinding – attaches the Role to the ServiceAccount
Token – credential used by the CI job

A rollout restart is implemented as a PATCH to the Deployment (spec.template.metadata.annotations), so the essential permission is:

verbs: ["patch"]

get is recommended; list is optional.

Kubernetes setup using YAML (recommended)

Create a single manifest with all required resources.

Full manifest

apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-deploy

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: gitlab-deploy-role
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "patch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: gitlab-deploy-binding
subjects:
  - kind: ServiceAccount
    name: gitlab-deploy
roleRef:
  kind: Role
  name: gitlab-deploy-role
  apiGroup: rbac.authorization.k8s.io

---
# Non-expiring token for CI/CD
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-deploy-token
  annotations:
    kubernetes.io/service-account.name: gitlab-deploy
type: kubernetes.io/service-account-token

Creating a non-expiring token (recommended for CI/CD)

Modern Kubernetes does not automatically create long-lived tokens. Instead, you explicitly create one via a Secret (as shown above).

Apply configuration

kubectl apply -f gitlab-deploy.yaml -n some-namespace

Retrieve the token

kubectl get secret gitlab-deploy-token -n some-namespace -o jsonpath="{.data.token}" | base64 -d

This token:

does not expire
is stable for CI/CD usage
is tied to the ServiceAccount lifecycle

This is the recommended approach for GitLab CI pipelines.

Kubernetes setup using CLI (alternative)

kubectl create serviceaccount gitlab-deploy -n some-namespace

kubectl create role gitlab-deploy-role \
  --verb=get,patch \
  --resource=deployments \
  -n some-namespace

kubectl create rolebinding gitlab-deploy-binding \
  --role=gitlab-deploy-role \
  --serviceaccount=some-namespace:gitlab-deploy \
  -n some-namespace

Temporary token (not recommended for CI)

kubectl create token gitlab-deploy -n some-namespace

This token:

expires (default ~1 hour)
is suitable only for testing

Obtaining GitLab environment variables

You need three variables in GitLab:

KUBE_TOKEN_DEV
KUBE_API_URL
KUBE_CA_PEM

Add them under Settings → CI/CD → Variables.

1. `KUBE_TOKEN_DEV`

For non-expiring token:

kubectl get secret gitlab-deploy-token -n some-namespace -o jsonpath="{.data.token}" | base64 -d

Copy and store as KUBE_TOKEN_DEV.

2. `KUBE_API_URL`

kubectl config view --minify -o jsonpath="{.clusters[0].cluster.server}"

Store output as KUBE_API_URL.

3. `KUBE_CA_PEM`

kubectl config view --raw -o jsonpath="{.clusters[0].cluster.certificate-authority-data}" | base64 -d

Store full certificate as KUBE_CA_PEM.

For k3s:

sudo cat /etc/rancher/k3s/k3s.yaml

GitLab CI configuration

`.gitlab-ci.yml`

deploy_dev:
  stage: deploy
  image: ubuntu:22.04

  before_script:
    - apt-get update && apt-get install -y curl
    - curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
    - install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

  script:
    - echo "$KUBE_CA_PEM" > ca.crt
    - kubectl config set-cluster k3s --server=$KUBE_API_URL --certificate-authority=ca.crt
    - kubectl config set-credentials gitlab --token=$KUBE_TOKEN_DEV
    - kubectl config set-context gitlab --cluster=k3s --user=gitlab --namespace=some-namespace
    - kubectl config use-context gitlab
    - kubectl rollout restart deployment/pod-deployment

  rules:
    - if: '$CI_COMMIT_BRANCH == "develop"'
      when: always
    - when: never

Explanation of the pipeline

Installs kubectl
Configures cluster endpoint and TLS
Authenticates using ServiceAccount token
Sets context with namespace
Executes rollout restart
Runs only on selected branch

Troubleshooting

RoleBinding error: cannot change roleRef

Fix:

kubectl delete rolebinding gitlab-deploy-binding -n some-namespace
kubectl apply -f gitlab-deploy.yaml

Forbidden errors

Check permissions:

kubectl auth can-i patch deployment \
  --as=system:serviceaccount:some-namespace:gitlab-deploy \
  -n some-namespace

Token expired

Cause:

Using kubectl create token

Fix:

Use Secret-based non-expiring token

Missing annotation warning

Safe to ignore when switching from create to apply.

API connection issues

Verify:

kubectl config view --minify

Upgrading K3s: Complete Guide

giveitatry — Wed, 01 Apr 2026 06:13:00 +0000

TL;DR

# Check current version
k3s --version

# Upgrade server node to latest stable (run on server first)
curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=stable sh -

# Upgrade to a specific version (upgrade One Minor Version at a Time)
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.32.5+k3s1 sh -

# Then upgrade each worker node (upgrade One Minor Version at a Time)
curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=stable K3S_URL=https://<server-ip>:6443 K3S_TOKEN=<token> sh -

# Verify
kubectl get nodes

Your Current Situation

You're running:

k3s version v1.28.4+k3s2 (6ba6c1b6)
go version go1.20.11

v1.28 reached end-of-life in late 2024. The current stable lines as of April 2026 are v1.32 and v1.33. That means you're roughly 4 minor versions behind, which is important — you cannot jump straight from v1.28 to v1.33 in one shot.

The Golden Rule: One Minor Version at a Time

When attempting to upgrade to a new version of K3s, the Kubernetes version skew policy applies. Ensure that your plan does not skip intermediate minor versions when upgrading.

This means your upgrade path from v1.28 must go:

v1.28 → v1.29 → v1.30 → v1.31 → v1.32 → v1.33

You cannot skip from v1.28 directly to v1.33. Each hop must be done separately, with the cluster verified healthy between each step.

Also: when upgrading, upgrade server nodes first one at a time, then any agent nodes. Never upgrade workers before the server.

Important: Traefik Version Change at v1.32

K3s versions v1.31 and earlier will install Traefik v2, while K3s versions v1.32 and later will install Traefik v3. If you're using Traefik as your ingress controller (which is the K3s default), the jump from v1.31 to v1.32 will upgrade Traefik from v2 to v3. Review the Traefik v3 migration guide before doing that hop if you have custom Traefik configuration or IngressRoute resources.

Important: etcd Warning for v1.34+

If you plan to go all the way to v1.34 or v1.35, there is a critical etcd requirement. The K3s project announced that there is no safe upgrade path from etcd 3.5 to 3.6 (included in v1.34+) unless you are running etcd v3.5.26 first. The January 2025 releases of K3s v1.32 include etcd v3.5.26, so upgrading to at least v1.32 before jumping to v1.34+ is mandatory to avoid cluster data issues.

Method 1: Manual Upgrade via Install Script (Recommended)

You can upgrade K3s by using the installation script, or by manually installing the binary of the desired version. The install script is the easiest approach.

Step 1 — Upgrade the server node, one minor version at a time

# SSH into your server node
ssh root@<server-ip>

# Hop 1: v1.28 → v1.29
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.29.13+k3s1 sh -

# Verify server is healthy before proceeding
kubectl get nodes
k3s --version

# Hop 2: v1.29 → v1.30
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.30.13+k3s1 sh -
kubectl get nodes

# Hop 3: v1.30 → v1.31
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.31.9+k3s1 sh -
kubectl get nodes

# Hop 4: v1.31 → v1.32 (Traefik v2 → v3 happens here!)
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.32.5+k3s1 sh -
kubectl get nodes

# Hop 5: v1.32 → v1.33 (optional, if you want latest stable)
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.33.1+k3s1 sh -
kubectl get nodes

Step 2 — Upgrade each worker node

After each server hop is confirmed healthy, upgrade the workers for that version before proceeding to the next hop:

# SSH into each worker node
ssh root@<worker-ip>

# Run the same version as the server just upgraded to
curl -sfL https://get.k3s.io | \
  INSTALL_K3S_VERSION=v1.29.13+k3s1 \
  K3S_URL=https://<server-ip>:6443 \
  K3S_TOKEN=<your-token> \
  sh -

# Verify worker rejoined
kubectl get nodes

Your token is in /var/lib/rancher/k3s/server/node-token on the server node:

cat /var/lib/rancher/k3s/server/node-token

Step 3 — Verify after each hop

# All nodes should show same version and Ready status
kubectl get nodes -o wide

# Check nothing is broken
kubectl get pods -A | grep -v Running | grep -v Completed

Method 2: Upgrade via Binary (Manual)

If you prefer more control, or the install script fails:

# Download the specific version binary
wget https://github.com/k3s-io/k3s/releases/download/v1.29.13%2Bk3s1/k3s

# Replace the binary
chmod +x k3s
sudo mv k3s /usr/local/bin/k3s

# Restart the service
sudo systemctl restart k3s        # on server
sudo systemctl restart k3s-agent  # on workers

# Verify
k3s --version

Draining Nodes Before Upgrading (For Production)

Containers for pods continue running even when K3s is stopped. If your workload is sensitive to brief API server outages, you should manually drain and cordon the node before restarting K3s, and uncordon it afterwards.

For a production cluster where you can't afford any disruption:

# Before upgrading a node, drain it from the server
kubectl drain <node-name> \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --force

# Run the upgrade on that node
# ...

# After the node is back up, uncordon it
kubectl uncordon <node-name>

After the Full Upgrade

Once all nodes are on the new version, do a full health check:

# All nodes Ready and on same version
kubectl get nodes -o wide

# No pods stuck in bad state
kubectl get pods -A | grep -v Running | grep -v Completed

# Check system pods specifically
kubectl get pods -n kube-system

# Verify cert status (especially useful since you're now in cert rotation territory)
sudo k3s certificate check

Summary

Your version	Target	Hops required
v1.28.4	v1.29	1
v1.28.4	v1.32	4 (go through 29, 30, 31, 32)
v1.28.4	v1.33	5 (go through 29, 30, 31, 32, 33)

Take it one minor version at a time, always server first then workers, verify healthy between each hop, and mind the Traefik v2→v3 change when crossing v1.32.

K3s Certificate Rotation: Complete Guide to Managing and Rotating Certificates

giveitatry — Wed, 01 Apr 2026 06:07:30 +0000

TL;DR — All Commands You Need

# 1. Check certificate status on any node
sudo k3s certificate check

# 2. Rotate on the SERVER node
sudo systemctl stop k3s
sudo k3s certificate rotate
sudo systemctl start k3s

# 3. Update kubeconfig after server rotation
sudo cp /etc/rancher/k3s/k3s.yaml $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# 4. IMMEDIATELY restart k3s-agent on EVERY worker node
#    (workers will disconnect after server cert rotation — this fixes it)
ssh root@<worker-ip>
sudo systemctl restart k3s-agent

# 5. Verify all nodes are back
kubectl get nodes -w

Certificate Types and Their Lifetimes

K3s manages two fundamentally different categories of certificates, and confusing them is the most common source of operational mistakes.

Client and Server Certificates — 365 days

K3s client and server certificates are valid for 365 days from their date of issuance. These are the leaf certificates used by every component in the cluster for mutual TLS: the API server, kubelet, etcd, kube-proxy, controller-manager, scheduler, and others. They are signed by the cluster's CA certificates.

The full list of rotatable services is: admin, api-server, controller-manager, scheduler, k3s-controller, k3s-server, cloud-controller, etcd, auth-proxy, kubelet, kube-proxy.

CA (Certificate Authority) Certificates — 10 years

By default, K3s generates self-signed CA certificates during startup of the first server node. These CA certificates are valid for 10 years from date of issuance, and are not automatically renewed. They live in /var/lib/rancher/k3s/server/tls and are the root of trust for the entire cluster. The authoritative copy is stored encrypted in the datastore (etcd or SQLite), with copies extracted to disk on startup.

Automatic Certificate Renewal — How It Works

Any certificates that are expired or within 120 days of expiring are automatically renewed every time K3s starts. This renewal reuses the existing keys, and extends the lifetime of the existing certificates.

This is the key distinction: automatic renewal extends the same keys, it doesn't rotate them. If you want brand new keys generated — for example after a security incident — you need the rotate subcommand covered below.

Version note: Prior to the May 2025 releases (v1.33.1+k3s1, v1.32.5+k3s1, v1.31.9+k3s1, v1.30.13+k3s1), alerts and rotation were triggered at 90 days instead of 120 days. If you're running an older cluster, the threshold is 90 days.

The Practical Implication

If your cluster nodes are rebooted or K3s is restarted at least once every few months as part of normal patching, certificates will silently renew themselves without any operator action. It is expected that you would be taking your hosts down periodically for patching and upgrading every few months — but reality has shown that many do not patch or reboot for more than 3 months, so the best practice is monitoring the certificate expiration.

A cluster that runs for an entire year without a restart will let its certificates expire.

Warning Signs — What You See Before Expiry

When a certificate is within 120 days of expiring, a Kubernetes Warning Event with reason: CertificateExpirationWarning is created, with a relation to the Node using the certificate.

You'll see events like this in your cluster logs:

Warning  CertificateExpirationWarning  Node certificates require attention -
restart k3s on this node to trigger automatic rotation:
kube-proxy/client-kube-proxy.crt: certificate CN=system:kube-proxy will
expire within 90 days at 2025-05-03T12:14:51Z

On newer versions these appear via:

kubectl get events -A --field-selector reason=CertificateExpirationWarning

What Happens When Certificates Actually Expire

This is where things get painful. You get something like this from systemctl status k3s:

x509: certificate has expired or is not yet valid

And any kubectl call returns:

Unable to connect to the server: x509: certificate has expired or is not yet valid

What still works: Apps that are currently running continue to run with no issues — workloads don't immediately crash. The data plane keeps operating. What breaks is the control plane: you lose kubectl access entirely, API server communication fails, and no new deployments, scaling, or config changes are possible. If nodes restart or pods crash, Kubernetes cannot reschedule them because the control plane is inaccessible.

The good news: certificates that are already expired are still automatically renewed every time K3s starts — so the recovery procedure is identical to the proactive rotation procedure. K3s is designed to self-heal on restart even in the fully expired state.

Checking Certificate Status

sudo k3s certificate check

Note: the --output table flag was added in the January 2025 releases (v1.32.0+k3s1, v1.31.5+k3s1, v1.30.9+k3s1) and will error on anything older. Check your version first:

k3s --version

# Only use --output table if you're on a January 2025+ release
sudo k3s certificate check --output table

Example output:

FILENAME                    SUBJECT                     USAGES       EXPIRES                  RESIDUAL TIME   STATUS
--------                    -------                     ------       -------                  -------------   ------
client-kube-proxy.crt       system:kube-proxy           ClientAuth   Jun 09, 2026 10:17 UTC   1 year          OK
client-kubelet.crt          system:node:k3s-server-1    ClientAuth   Jun 09, 2026 10:17 UTC   1 year          OK
serving-kubelet.crt         k3s-server-1                ServerAuth   Jun 09, 2026 10:17 UTC   1 year          OK
client-k3s-controller.crt   system:k3s-controller       ClientAuth   Jun 09, 2026 10:17 UTC   1 year          OK

Each certificate file in the FILENAME column contains at least two certificates — the leaf (or end entity) client/server certificate, any intermediate Certificate Authority certificates, and the root Certificate Authority certificate. That's why you see each filename appear twice — the leaf cert at 1 year, and the CA cert at 10 years.

To inspect a specific cert directly with openssl:

sudo openssl x509 \
  -in /var/lib/rancher/k3s/server/tls/client-admin.crt \
  -noout -dates

To inspect the certificate currently embedded in your kubeconfig:

sudo cat /etc/rancher/k3s/k3s.yaml \
  | grep 'client-certificate-data' \
  | awk '{print $2}' \
  | base64 -d \
  | openssl x509 -noout -dates

This is useful to confirm your local kubectl context is using up-to-date credentials, separate from what the node itself reports.

Manual Rotation — Step by Step

On the Server Node

# Step 1: Stop K3s
sudo systemctl stop k3s

# Step 2: Rotate certificates (generates new certs AND new keys)
sudo k3s certificate rotate

# Step 3: Start K3s
sudo systemctl start k3s

After K3s starts back up, the new certificate is written to /etc/rancher/k3s/k3s.yaml. This file is your kubeconfig and now contains the updated client certificate, client key, and certificate authority data:

cd /etc/rancher/k3s
ls
# k3s.yaml

cat k3s.yaml

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <base64-encoded-new-CA>
    server: https://127.0.0.1:6443
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
users:
- name: default
  user:
    client-certificate-data: <base64-encoded-new-cert>
    client-key-data: <base64-encoded-new-key>

# Step 4: Copy the updated kubeconfig
sudo cp /etc/rancher/k3s/k3s.yaml $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# If managing the cluster from a remote machine, copy it there too
scp root@<server-ip>:/etc/rancher/k3s/k3s.yaml ~/.kube/config
# Fix the server address if it points to 127.0.0.1
sed -i 's/127.0.0.1/<your-server-ip>/g' ~/.kube/config

# Step 5: Verify server is healthy
sudo k3s certificate check
kubectl get nodes

Any CI/CD pipelines, Helm deployments, or developer workstations that had a copy of the old kubeconfig will also need this updated file — the old client certificate is invalid the moment rotation runs.

Rotating Specific Services Only

You can limit rotation to specific components rather than rotating everything at once:

sudo k3s certificate rotate --service api-server,kubelet
sudo k3s certificate rotate --service etcd
sudo k3s certificate rotate --service admin,controller-manager,scheduler

This is useful when only certain certs are approaching expiry, or when you want to minimize the blast radius in a sensitive environment. The stop/start of K3s is still required around this command.

⚠️ Warning: Worker Nodes Will Disconnect After Rotation

This is something the official documentation doesn't warn you about clearly enough, and it can cause a cascading failure that looks far worse than a simple cert issue.

What Happens

When you rotate certificates on the server node, the API server starts presenting new certificates. Worker nodes running k3s-agent were trusting the old certificates. Until the agent is restarted and picks up the new trust chain, kubelet on each worker loses the ability to communicate with the API server:

Kubelet stopped posting node status.

The workers show as NotReady even though the machines themselves are perfectly healthy.

The Cascade With Local Storage

If you're using local-path storage (the K3s default), this gets significantly worse:

You rotate certs on the server 🔄
Kubelet on workers stops trusting the API server ❌
Workers become NotReady ❌
PVCs using local-path are physically locked to those dead nodes ❌
Pods tied to that storage cannot reschedule ❌
Scheduler starts throwing volume node affinity conflict ❌
Any stateful workload (databases, etc.) is now stuck in Pending or Initializing forever ❌

The fix is straightforward — but you have to do it immediately after rotating the server, before anything starts crashing and trying to reschedule.

Fix — Restart the Agent on Every Worker Node

SSH into each worker and restart the agent:

# Step 1 — SSH into the worker
ssh root@<worker-ip>

# Step 2 — Restart k3s-agent
sudo systemctl restart k3s-agent

# Step 3 — Check status
sudo systemctl status k3s-agent

If the agent still won't come back after a restart:

# Full reboot as last resort
sudo reboot

Then verify from the control plane that all nodes are back:

kubectl get nodes -w

Wait for all workers to flip back to Ready. Once they do, any stuck pods will reschedule automatically — no manual intervention on the workloads themselves is needed.

The Correct Order of Operations for a Full Cluster Rotation

To avoid the disconnection cascade entirely, always treat cert rotation as a coordinated cluster-wide operation:

1. Stop k3s on server           →  sudo systemctl stop k3s
2. Rotate certs on server       →  sudo k3s certificate rotate
3. Start k3s on server          →  sudo systemctl start k3s
4. Update kubeconfig            →  cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
5. Restart agent on each worker →  sudo systemctl restart k3s-agent
6. Verify all nodes Ready       →  kubectl get nodes

Don't wait between steps 4 and 5. The longer workers run without restarting their agent, the higher the chance that a pod crash or node pressure event triggers a rescheduling attempt that hits the volume node affinity conflict wall.

CA Certificate Rotation

CA rotation is a completely different operation from leaf cert rotation. To rotate CA certificates and keys, use the k3s certificate rotate-ca command. The command performs integrity checks to confirm that the updated certificates and keys are usable. If the updated data is acceptable, the datastore's encrypted bootstrap key is updated, and the new certificates and keys will be used the next time K3s starts. If problems are encountered while validating the certificates and keys, an error is reported to the system log and the operation is cancelled without changes.

Non-disruptive vs. Disruptive Rotation

A cluster that has been started with custom CA certificates can renew or rotate the CA certificates and keys non-disruptively, as long as the same root CA is used. If a new root CA is required, the rotation will be disruptive. The k3s certificate rotate-ca --force option must be used, all nodes that were joined with a secure token (including servers) will need to be reconfigured to use the new token value, and pods will need to be restarted to trust the new root CA.

The safe path is to generate new intermediate CAs signed by the existing root CA — existing trust chains remain valid, nodes can rejoin without token changes, and pods don't need restarts.

CA Rotation Procedure

# Stage new CA certificates into a TEMP directory
# Do NOT place them directly into the live /server/tls directory
mkdir -p /tmp/k3s-ca-rotate

# Use the helper script from the K3s repo to generate new CAs
curl -o /tmp/generate-custom-ca-certs.sh \
  https://raw.githubusercontent.com/k3s-io/k3s/main/contrib/util/generate-custom-ca-certs.sh
chmod +x /tmp/generate-custom-ca-certs.sh
K3S_DATADIR=/tmp/k3s-ca-rotate /tmp/generate-custom-ca-certs.sh

# Load the new CAs into the datastore
sudo k3s certificate rotate-ca --path /tmp/k3s-ca-rotate/server/tls

# Restart K3s on all nodes — servers first, then agents
sudo systemctl restart k3s

Back up the generated root and intermediate CA files somewhere safe after this — you'll need them if you ever need to rotate CAs again in the future.

Monitoring — Preventing Surprise Expiry

Watch for Kubernetes warning events:

kubectl get events -A \
  --field-selector reason=CertificateExpirationWarning \
  --watch

Prometheus alert rule (works with kube-prometheus-stack):

- alert: K3sCertificateExpiringSoon
  expr: |
    (k3s_certificate_expiry_seconds - time()) < 30 * 24 * 3600
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "K3s certificate expiring in less than 30 days on {{ $labels.node }}"

Simple weekly cron check on the server node:

#!/bin/bash
# /etc/cron.weekly/k3s-cert-check
k3s certificate check | mail -s "K3s cert check on $(hostname)" ops@yourcompany.com

Summary Table

Certificate type	Validity	Auto-renewal	Rotation command
Client/Server leaf certs	365 days	Yes, on restart (if < 120 days remaining)	`k3s certificate rotate`
CA certificates	10 years	No	`k3s certificate rotate-ca`
kubeconfig embedded cert	Same as leaf	Follows leaf rotation	Copy updated `k3s.yaml`

Key Takeaways

Restart K3s regularly. Even just for patching every few months — this is what triggers automatic certificate renewal. A cluster that never restarts will expire its certs at the 1-year mark.
rotate ≠ restart. A plain restart extends existing keys. k3s certificate rotate generates entirely new keys. Use rotate when you need cryptographic freshness, not just extended validity.
Always restart worker agents immediately after rotating the server. Workers disconnect the moment the server starts presenting new certs. Don't wait — do it before anything tries to reschedule.
local-path storage amplifies worker disconnection. PVCs are locked to specific nodes. If a worker goes NotReady while holding a PVC, any pod using that storage is stuck until the worker comes back.
Update your kubeconfig after server node rotation — the embedded client certificate changes, and any CI/CD pipelines using the old kubeconfig will break immediately.
CA certs are your 10-year bomb. Mark a calendar reminder. They won't warn you and won't auto-renew.
Backups are automatic. Old certs are saved to /var/lib/rancher/k3s/server/tls-<timestamp> before rotation, giving you a rollback path.

Running a TeamCity Agent as a systemd Service

giveitatry — Tue, 24 Mar 2026 06:44:39 +0000

Running a TeamCity agent as a system service ensures it continues working after logout and automatically starts on boot. This is the recommended production setup.

📁 Prerequisites

Make sure your TeamCity agent is installed in:

/opt/teamcity/bin/agent.sh

👉 The agent.sh script must exist inside the bin directory:

/opt/teamcity/bin/agent.sh

Also ensure it is executable:

chmod +x /opt/teamcity/bin/agent.sh

⚙️ Create systemd service

Create the file:

sudo nano /etc/systemd/system/teamcity-agent.service

Paste the following configuration:

[Unit]
Description=TeamCity Build Agent
After=network.target

[Service]
Type=oneshot
User=root
Group=root
WorkingDirectory=/opt/teamcity

ExecStart=/opt/teamcity/bin/agent.sh start
ExecStop=/opt/teamcity/bin/agent.sh stop

RemainAfterExit=yes
SuccessExitStatus=0 143

[Install]
WantedBy=multi-user.target

🚀 Enable and start the service

Reload systemd and start the agent:

sudo systemctl daemon-reload
sudo systemctl enable teamcity-agent
sudo systemctl start teamcity-agent

🔍 Verify status

systemctl status teamcity-agent

Expected result:

Active: active (exited)

This is normal for TeamCity agents when using Type=oneshot.

⚠️ Notes

The agent must be located in /opt/teamcity/bin/agent.sh
RemainAfterExit=yes is required so systemd considers the service active
SuccessExitStatus=143 prevents false failure detection
Using root works, but for production it's safer to use a dedicated user

Kafka 4.2.0 on Kubernetes - Complete Setup Guide - Exposed to Internet

giveitatry — Mon, 23 Mar 2026 08:20:43 +0000

3-broker Kafka cluster on k3s with KRaft, SASL/SCRAM, and external access via Traefik.

Before you start — get your Traefik IP:

kubectl get svc traefik -n kube-system

Copy any IP from the EXTERNAL-IP column. Replace every occurrence of
192.168.1.119 in the YAMLs below with your actual IP.

Stage 1 — Bootstrap

Apply all of these files in order. This stage starts the cluster with port 9094
open (no auth) so you can register SCRAM credentials.

1.1 namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: kafka

kubectl apply -f namespace.yaml

1.2 kafka-jaas.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-jaas
  namespace: kafka
data:
  jaas.conf: |
    KafkaServer {
      org.apache.kafka.common.security.scram.ScramLoginModule required
      username="admin"
      password="supersecret";
    };

kubectl apply -f kafka-jaas.yaml

1.3 kafka-sasl.yaml

apiVersion: v1
kind: Secret
metadata:
  name: kafka-sasl
  namespace: kafka
type: Opaque
stringData:
  username: admin
  password: supersecret

kubectl apply -f kafka-sasl.yaml

1.4 traefik-config.yaml

Opens three TCP entrypoints on Traefik — one per broker.

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    ports:
      kafka-0:
        port: 9092
        expose:
          default: true
        exposedPort: 9092
        protocol: TCP
      kafka-1:
        port: 9093
        expose:
          default: true
        exposedPort: 9093
        protocol: TCP
      kafka-2:
        port: 9094
        expose:
          default: true
        exposedPort: 9094
        protocol: TCP

kubectl apply -f traefik-config.yaml
kubectl rollout status deployment/traefik -n kube-system

Verify ports appeared:

kubectl get svc traefik -n kube-system
# PORT(S) should include 9092, 9093, 9094 alongside 80 and 443

1.5 kafka-traefik.yaml

Headless service for internal pod DNS, one ClusterIP service per broker
(Traefik v3 routes to port 9095 on each pod), and IngressRouteTCP rules.

apiVersion: v1
kind: Service
metadata:
  name: kafka-headless
  namespace: kafka
spec:
  clusterIP: None
  selector:
    app: kafka
  ports:
    - name: internal
      port: 9092
    - name: controller
      port: 9093
    - name: plaintext-bootstrap
      port: 9094

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-0-external
  namespace: kafka
spec:
  type: ClusterIP
  selector:
    app: kafka
    statefulset.kubernetes.io/pod-name: kafka-0
  ports:
    - name: external
      port: 9092
      targetPort: 9095

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-1-external
  namespace: kafka
spec:
  type: ClusterIP
  selector:
    app: kafka
    statefulset.kubernetes.io/pod-name: kafka-1
  ports:
    - name: external
      port: 9092
      targetPort: 9095

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-2-external
  namespace: kafka
spec:
  type: ClusterIP
  selector:
    app: kafka
    statefulset.kubernetes.io/pod-name: kafka-2
  ports:
    - name: external
      port: 9092
      targetPort: 9095

---
apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
  name: kafka-0-tcp
  namespace: kafka
spec:
  entryPoints:
    - kafka-0
  routes:
    - match: HostSNI(`*`)
      services:
        - name: kafka-0-external
          port: 9092

---
apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
  name: kafka-1-tcp
  namespace: kafka
spec:
  entryPoints:
    - kafka-1
  routes:
    - match: HostSNI(`*`)
      services:
        - name: kafka-1-external
          port: 9092

---
apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
  name: kafka-2-tcp
  namespace: kafka
spec:
  entryPoints:
    - kafka-2
  routes:
    - match: HostSNI(`*`)
      services:
        - name: kafka-2-external
          port: 9092

kubectl apply -f kafka-traefik.yaml

1.6 kafka-stateful-bootstrap.yaml

⚠️ Replace 192.168.1.119 with your Traefik IP before applying.

Includes:

EXTERNAL listener on port 9095 — Traefik routes external traffic here
PLAINTEXT listener on port 9094 — temporary, no auth, used to register SCRAM credentials
init container calculates the external port per broker: kafka-0 → 9092, kafka-1 → 9093, kafka-2 → 9094

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
  namespace: kafka
spec:
  serviceName: kafka-headless
  replicas: 3
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      securityContext:
        fsGroup: 1001
      volumes:
        - name: kafka-config
          emptyDir: {}
        - name: kafka-jaas
          configMap:
            name: kafka-jaas
      initContainers:
        - name: init-node-id
          image: busybox:1.36
          command:
            - sh
            - -c
            - |
              ORDINAL=$(hostname | awk -F'-' '{print $NF}')
              echo "$ORDINAL" > /config/node-id
              EXTERNAL_PORT=$((9092 + ORDINAL))
              echo "$EXTERNAL_PORT" > /config/external-port
          volumeMounts:
            - name: kafka-config
              mountPath: /config
        - name: format-storage
          image: apache/kafka:4.2.0
          command:
            - sh
            - -c
            - |
              NODE_ID=$(cat /config/node-id)
              if [ ! -f "/data/meta.properties" ]; then
                echo "Formatting storage for node $NODE_ID..."
                echo "node.id=$NODE_ID" > /tmp/kraft.properties
                echo "process.roles=broker,controller" >> /tmp/kraft.properties
                echo "controller.quorum.voters=0@kafka-0.kafka-headless.kafka.svc.cluster.local:9093,1@kafka-1.kafka-headless.kafka.svc.cluster.local:9093,2@kafka-2.kafka-headless.kafka.svc.cluster.local:9093" >> /tmp/kraft.properties
                echo "listeners=PLAINTEXT://:9092,CONTROLLER://:9093" >> /tmp/kraft.properties
                echo "advertised.listeners=PLAINTEXT://localhost:9092" >> /tmp/kraft.properties
                echo "listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT" >> /tmp/kraft.properties
                echo "inter.broker.listener.name=PLAINTEXT" >> /tmp/kraft.properties
                echo "controller.listener.names=CONTROLLER" >> /tmp/kraft.properties
                echo "log.dirs=/data" >> /tmp/kraft.properties
                /opt/kafka/bin/kafka-storage.sh format \
                  --ignore-formatted \
                  --cluster-id q1Sh-9_ISia_zwGINzRvyQ \
                  --config /tmp/kraft.properties
              else
                echo "Already formatted, skipping."
              fi
          volumeMounts:
            - name: kafka-data
              mountPath: /data
            - name: kafka-config
              mountPath: /config
      containers:
        - name: kafka
          image: apache/kafka:4.2.0
          command:
            - sh
            - -c
            - |
              export KAFKA_NODE_ID=$(cat /config/node-id)
              export EXTERNAL_PORT=$(cat /config/external-port)
              export KAFKA_ADVERTISED_LISTENERS="INTERNAL://$(POD_NAME).kafka-headless.kafka.svc.cluster.local:9092,EXTERNAL://192.168.1.119:${EXTERNAL_PORT},PLAINTEXT://$(POD_NAME).kafka-headless.kafka.svc.cluster.local:9094"
              exec /etc/kafka/docker/run
          ports:
            - containerPort: 9092
            - containerPort: 9093
            - containerPort: 9094
            - containerPort: 9095
          env:
            - name: CLUSTER_ID
              value: "q1Sh-9_ISia_zwGINzRvyQ"
            - name: KAFKA_PROCESS_ROLES
              value: "broker,controller"
            - name: KAFKA_CONTROLLER_LISTENER_NAMES
              value: "CONTROLLER"
            - name: KAFKA_CONTROLLER_QUORUM_VOTERS
              value: "0@kafka-0.kafka-headless.kafka.svc.cluster.local:9093,1@kafka-1.kafka-headless.kafka.svc.cluster.local:9093,2@kafka-2.kafka-headless.kafka.svc.cluster.local:9093"
            - name: KAFKA_LISTENERS
              value: "INTERNAL://:9092,CONTROLLER://:9093,EXTERNAL://:9095,PLAINTEXT://:9094"
            - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
              value: "INTERNAL:SASL_PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:SASL_PLAINTEXT,PLAINTEXT:PLAINTEXT"
            - name: KAFKA_INTER_BROKER_LISTENER_NAME
              value: "INTERNAL"
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: KAFKA_SASL_ENABLED_MECHANISMS
              value: SCRAM-SHA-512
            - name: KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL
              value: SCRAM-SHA-512
            - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
              value: "3"
            - name: KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR
              value: "3"
            - name: KAFKA_TRANSACTION_STATE_LOG_MIN_ISR
              value: "2"
            - name: KAFKA_MIN_INSYNC_REPLICAS
              value: "2"
            - name: KAFKA_LOG_DIRS
              value: /data
            - name: KAFKA_OPTS
              value: "-Djava.security.auth.login.config=/opt/kafka/config/jaas/jaas.conf"
          volumeMounts:
            - name: kafka-data
              mountPath: /data
            - name: kafka-config
              mountPath: /config
            - name: kafka-jaas
              mountPath: /opt/kafka/config/jaas
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: "2"
              memory: 4Gi
  volumeClaimTemplates:
    - metadata:
        name: kafka-data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi

kubectl apply -f kafka-stateful-bootstrap.yaml
kubectl rollout status statefulset/kafka -n kafka

1.7 Register SCRAM credentials

Port 9094 is open and unauthenticated. Use it to write the admin user into
Kafka's metadata store. This is a one-time operation.

kubectl exec -n kafka kafka-0 -- \
  /opt/kafka/bin/kafka-configs.sh \
  --bootstrap-server kafka-0.kafka-headless.kafka.svc.cluster.local:9094 \
  --alter \
  --add-config 'SCRAM-SHA-512=[password=supersecret]' \
  --entity-type users \
  --entity-name admin

Expected output:

Completed updating config for user admin.

To add more users (application service accounts), repeat the command with
different --entity-name and password values while port 9094 is still open.

Stage 2 — Production (close port 9094)

Port 9094 is now unnecessary and a security risk. Apply the final StatefulSet
to remove it. No other files change.

2.1 kafka-stateful-final.yaml

⚠️ Replace 192.168.1.119 with your Traefik IP before applying.

Identical to the bootstrap version except:

PLAINTEXT://:9094 removed from KAFKA_LISTENERS
PLAINTEXT:PLAINTEXT removed from KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
PLAINTEXT entry removed from KAFKA_ADVERTISED_LISTENERS
containerPort: 9094 removed

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
  namespace: kafka
spec:
  serviceName: kafka-headless
  replicas: 3
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      securityContext:
        fsGroup: 1001
      volumes:
        - name: kafka-config
          emptyDir: {}
        - name: kafka-jaas
          configMap:
            name: kafka-jaas
      initContainers:
        - name: init-node-id
          image: busybox:1.36
          command:
            - sh
            - -c
            - |
              ORDINAL=$(hostname | awk -F'-' '{print $NF}')
              echo "$ORDINAL" > /config/node-id
              EXTERNAL_PORT=$((9092 + ORDINAL))
              echo "$EXTERNAL_PORT" > /config/external-port
          volumeMounts:
            - name: kafka-config
              mountPath: /config
        - name: format-storage
          image: apache/kafka:4.2.0
          command:
            - sh
            - -c
            - |
              NODE_ID=$(cat /config/node-id)
              if [ ! -f "/data/meta.properties" ]; then
                echo "Formatting storage for node $NODE_ID..."
                echo "node.id=$NODE_ID" > /tmp/kraft.properties
                echo "process.roles=broker,controller" >> /tmp/kraft.properties
                echo "controller.quorum.voters=0@kafka-0.kafka-headless.kafka.svc.cluster.local:9093,1@kafka-1.kafka-headless.kafka.svc.cluster.local:9093,2@kafka-2.kafka-headless.kafka.svc.cluster.local:9093" >> /tmp/kraft.properties
                echo "listeners=PLAINTEXT://:9092,CONTROLLER://:9093" >> /tmp/kraft.properties
                echo "advertised.listeners=PLAINTEXT://localhost:9092" >> /tmp/kraft.properties
                echo "listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT" >> /tmp/kraft.properties
                echo "inter.broker.listener.name=PLAINTEXT" >> /tmp/kraft.properties
                echo "controller.listener.names=CONTROLLER" >> /tmp/kraft.properties
                echo "log.dirs=/data" >> /tmp/kraft.properties
                /opt/kafka/bin/kafka-storage.sh format \
                  --ignore-formatted \
                  --cluster-id q1Sh-9_ISia_zwGINzRvyQ \
                  --config /tmp/kraft.properties
              else
                echo "Already formatted, skipping."
              fi
          volumeMounts:
            - name: kafka-data
              mountPath: /data
            - name: kafka-config
              mountPath: /config
      containers:
        - name: kafka
          image: apache/kafka:4.2.0
          command:
            - sh
            - -c
            - |
              export KAFKA_NODE_ID=$(cat /config/node-id)
              export EXTERNAL_PORT=$(cat /config/external-port)
              export KAFKA_ADVERTISED_LISTENERS="INTERNAL://$(POD_NAME).kafka-headless.kafka.svc.cluster.local:9092,EXTERNAL://192.168.1.119:${EXTERNAL_PORT}"
              exec /etc/kafka/docker/run
          ports:
            - containerPort: 9092
            - containerPort: 9093
            - containerPort: 9095
          env:
            - name: CLUSTER_ID
              value: "q1Sh-9_ISia_zwGINzRvyQ"
            - name: KAFKA_PROCESS_ROLES
              value: "broker,controller"
            - name: KAFKA_CONTROLLER_LISTENER_NAMES
              value: "CONTROLLER"
            - name: KAFKA_CONTROLLER_QUORUM_VOTERS
              value: "0@kafka-0.kafka-headless.kafka.svc.cluster.local:9093,1@kafka-1.kafka-headless.kafka.svc.cluster.local:9093,2@kafka-2.kafka-headless.kafka.svc.cluster.local:9093"
            - name: KAFKA_LISTENERS
              value: "INTERNAL://:9092,CONTROLLER://:9093,EXTERNAL://:9095"
            - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
              value: "INTERNAL:SASL_PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:SASL_PLAINTEXT"
            - name: KAFKA_INTER_BROKER_LISTENER_NAME
              value: "INTERNAL"
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: KAFKA_SASL_ENABLED_MECHANISMS
              value: SCRAM-SHA-512
            - name: KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL
              value: SCRAM-SHA-512
            - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
              value: "3"
            - name: KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR
              value: "3"
            - name: KAFKA_TRANSACTION_STATE_LOG_MIN_ISR
              value: "2"
            - name: KAFKA_MIN_INSYNC_REPLICAS
              value: "2"
            - name: KAFKA_LOG_DIRS
              value: /data
            - name: KAFKA_OPTS
              value: "-Djava.security.auth.login.config=/opt/kafka/config/jaas/jaas.conf"
          volumeMounts:
            - name: kafka-data
              mountPath: /data
            - name: kafka-config
              mountPath: /config
            - name: kafka-jaas
              mountPath: /opt/kafka/config/jaas
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: "2"
              memory: 4Gi
  volumeClaimTemplates:
    - metadata:
        name: kafka-data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi

kubectl apply -f kafka-stateful-final.yaml
kubectl rollout status statefulset/kafka -n kafka

2.2 Verify

kubectl exec -n kafka kafka-0 -- \
  /opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-controller kafka-0.kafka-headless.kafka.svc.cluster.local:9093 \
  describe --status

Healthy output:

LeaderId:             0
MaxFollowerLag:       0
MaxFollowerLagTimeMs: 0
CurrentVoters:        [{"id": 0, ...}, {"id": 1, ...}, {"id": 2, ...}]
CurrentObservers:     []

2.3 Python client

pip install kafka-python-ng

from kafka import KafkaProducer, KafkaConsumer, KafkaAdminClient
from kafka.admin import NewTopic
from kafka.errors import TopicAlreadyExistsError
import json, time

# Replace 192.168.1.119 with your Traefik IP
BOOTSTRAP_SERVERS = [
    "192.168.1.119:9092",  # kafka-0
    "192.168.1.119:9093",  # kafka-1
    "192.168.1.119:9094",  # kafka-2
]

SASL_CONFIG = {
    "security_protocol": "SASL_PLAINTEXT",
    "sasl_mechanism": "SCRAM-SHA-512",
    "sasl_plain_username": "admin",
    "sasl_plain_password": "supersecret",
}

TOPIC = "orders"


def create_topic(topic):
    admin = KafkaAdminClient(bootstrap_servers=BOOTSTRAP_SERVERS, **SASL_CONFIG)
    try:
        admin.create_topics([
            NewTopic(name=topic, num_partitions=3, replication_factor=3)
        ])
        print(f"Topic '{topic}' created.")
    except TopicAlreadyExistsError:
        print(f"Topic '{topic}' already exists.")
    finally:
        admin.close()


def produce(topic=TOPIC, count=10):
    producer = KafkaProducer(
        bootstrap_servers=BOOTSTRAP_SERVERS,
        **SASL_CONFIG,
        value_serializer=lambda v: json.dumps(v).encode(),
        key_serializer=lambda k: k.encode() if k else None,
        acks="all",
    )
    for i in range(count):
        meta = producer.send(
            topic,
            key=str(i),
            value={"id": i, "ts": time.time()}
        ).get(timeout=10)
        print(f"  sent [{i}] partition={meta.partition} offset={meta.offset}")
    producer.flush()
    producer.close()


def consume(topic=TOPIC):
    consumer = KafkaConsumer(
        topic,
        bootstrap_servers=BOOTSTRAP_SERVERS,
        **SASL_CONFIG,
        value_deserializer=lambda v: json.loads(v.decode()),
        key_deserializer=lambda k: k.decode() if k else None,
        group_id="my-group",
        auto_offset_reset="earliest",
        consumer_timeout_ms=5000,
    )
    for msg in consumer:
        print(
            f"  recv key={msg.key} "
            f"partition={msg.partition} "
            f"offset={msg.offset} "
            f"value={msg.value}"
        )
    consumer.close()


if __name__ == "__main__":
    create_topic(TOPIC)
    produce()
    consume()

Troubleshooting

Pods not starting

kubectl logs -n kafka kafka-0 -c format-storage
kubectl logs -n kafka kafka-0 -c init-node-id

SCRAM registration times out

Port 9094 is not reachable. Check the headless service includes it:

kubectl get svc kafka-headless -n kafka

Python client connects but gets wrong broker addresses

The Traefik IP in the StatefulSet command block is wrong. Fix it and roll:

kubectl rollout restart statefulset/kafka -n kafka

Traefik ports not appearing

kubectl rollout restart deployment/traefik -n kube-system
kubectl get svc traefik -n kube-system

Deploying Apache Kafka 4.2.0 on Kubernetes with KRaft, SASL, and High Availability

giveitatry — Sun, 22 Mar 2026 12:41:13 +0000

Overview

This guide walks through deploying a production-grade Apache Kafka cluster on Kubernetes using KRaft mode (no ZooKeeper), SASL/SCRAM-SHA-512 authentication, and a 3-node StatefulSet for high availability. It covers every manifest file required, the reasoning behind each configuration decision, the SCRAM credential bootstrap process, common pitfalls encountered in practice, and the steps needed to take the cluster from running to production-ready.

Prerequisites

Tools

kubectl configured against your target cluster
A Kubernetes cluster with at least 3 nodes (one per Kafka pod) with sufficient resources
Persistent volume provisioner available (e.g. local-path, Longhorn, Ceph, AWS EBS)
keytool (part of the JDK) if you plan to add TLS later

Cluster Resources

Each Kafka pod in this guide requests 500m CPU and 1Gi RAM, with limits of 2 CPU and 4Gi RAM. For production you should size these based on your throughput requirements. A minimum of 10Gi persistent storage per broker is configured.

Kubernetes Version

Kubernetes 1.21 or later is recommended. StatefulSets, headless Services, and ConfigMaps used in this guide are stable APIs available in all modern versions.

Architecture

The cluster consists of three pods (kafka-0, kafka-1, kafka-2) each running in combined broker+controller mode using KRaft. This means each pod participates in the Raft quorum for metadata management as well as serving producer and consumer traffic.

┌─────────────────────────────────────────┐
│             Kafka Namespace             │
│                                         │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐ │
│  │ kafka-0 │  │ kafka-1 │  │ kafka-2 │ │
│  │ broker  │  │ broker  │  │ broker  │ │
│  │ + ctrl  │  │ + ctrl  │  │ + ctrl  │ │
│  └────┬────┘  └────┬────┘  └────┬────┘ │
│       │             │             │     │
│  port 9092 (SASL_PLAINTEXT — broker)    │
│  port 9093 (PLAINTEXT — controller)     │
│                                         │
│  ┌─────────────────────────────────┐    │
│  │   kafka-headless (ClusterIP:    │    │
│  │   None) — DNS per pod           │    │
│  └─────────────────────────────────┘    │
└─────────────────────────────────────────┘

KRaft requires a majority quorum (2 out of 3) to elect a leader and commit metadata. The cluster can tolerate the loss of one node.

File Structure

You need five manifest files plus one temporary modification during the SCRAM bootstrap process:

namespace.yaml           — Kafka namespace
kafka-svc.yaml           — Headless service for pod DNS
kafka-jaas.yaml          — JAAS config for inter-broker SCRAM auth
kafka-sasl.yaml          — Secret holding admin credentials for client apps
kafka-stateful-set.yaml  — Brokers, init containers, volumes, config

Step 1 — Namespace

# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: kafka

Apply first so all subsequent resources land in the correct namespace:

kubectl apply -f namespace.yaml

Step 2 — Headless Service

# kafka-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: kafka-headless
  namespace: kafka
spec:
  clusterIP: None
  selector:
    app: kafka
  ports:
    - name: internal
      port: 9092
    - name: controller
      port: 9093

A headless service (clusterIP: None) causes Kubernetes DNS to create per-pod A records:

kafka-0.kafka-headless.kafka.svc.cluster.local
kafka-1.kafka-headless.kafka.svc.cluster.local
kafka-2.kafka-headless.kafka.svc.cluster.local

These stable DNS names are what the KRaft quorum voters list and the advertised listeners are built from. They survive pod restarts and rescheduling because they are tied to the StatefulSet ordinal, not the pod IP.

Step 3 — JAAS Configuration

# kafka-jaas.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-jaas
  namespace: kafka
data:
  jaas.conf: |
    KafkaServer {
      org.apache.kafka.common.security.scram.ScramLoginModule required
      username="admin"
      password="supersecret";
    };

This file configures the Java Authentication and Authorization Service (JAAS) for the Kafka broker process itself. The username and password here are the inter-broker credentials — what each broker uses when authenticating to other brokers on the INTERNAL listener.

Important: JAAS alone does not create the SCRAM user in Kafka's metadata store. That is a separate bootstrap step performed after the cluster is running (see Step 7). The JAAS file tells the broker process what credentials to use, but those credentials must also exist in Kafka's internal metadata before inter-broker authentication will succeed.

Step 4 — SASL Secret

# kafka-sasl.yaml
apiVersion: v1
kind: Secret
metadata:
  name: kafka-sasl
  namespace: kafka
type: Opaque
stringData:
  username: admin
  password: supersecret

This secret can be mounted into client pods or referenced by applications that need to connect to Kafka. It is separate from the JAAS ConfigMap so that client credentials can be managed independently of the broker configuration.

Step 5 — StatefulSet

This is the core manifest. It contains two init containers and one main broker container.

Why Two Init Containers?

init-node-id runs a tiny busybox container to extract the pod ordinal (0, 1, or 2) from the hostname and writes it to a shared emptyDir volume. This is necessary because Kubernetes environment variable substitution does not support string manipulation, so you cannot derive the integer 0 from the pod name kafka-0 using env vars alone.

format-storage runs the actual Kafka image to format the KRaft log directory using kafka-storage.sh. It only runs if /data/meta.properties does not already exist, making it safe on pod restarts. The temporary kraft.properties file used during formatting includes a dummy plaintext broker listener — this is required because Kafka's config validator refuses to proceed if the only listener defined is a controller listener. These values are only used during formatting and are not used by the running broker.

Why `/etc/kafka/docker/run` Instead of `kafka-server-start.sh`?

The official apache/kafka Docker image ships an entrypoint at /etc/kafka/docker/run that translates KAFKA_* environment variables into server.properties entries before starting the broker. Calling kafka-server-start.sh directly bypasses this translation entirely — env vars like KAFKA_LOG_DIRS are ignored and the broker falls back to compiled-in defaults, causing it to look for data in the wrong directory.

Why `CLUSTER_ID` as an Env Var?

If CLUSTER_ID is not set, the official entrypoint generates a new random cluster ID on every pod start and attempts to re-format storage. This fails because the existing meta.properties already has a different ID written by the init container. Always set CLUSTER_ID to match the value used during the format step.

Generating Your Own Cluster ID

The cluster ID q1Sh-9_ISia_zwGINzRvyQ used in this guide was generated with:

/opt/kafka/bin/kafka-storage.sh random-uuid

Generate your own unique ID for each cluster you deploy. Do not reuse IDs across clusters.

# kafka-stateful-set.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
  namespace: kafka
spec:
  serviceName: kafka-headless
  replicas: 3

  selector:
    matchLabels:
      app: kafka

  template:
    metadata:
      labels:
        app: kafka

    spec:
      securityContext:
        fsGroup: 1001

      volumes:
        - name: kafka-config
          emptyDir: {}

        - name: kafka-jaas
          configMap:
            name: kafka-jaas

      initContainers:

        - name: init-node-id
          image: busybox:1.36
          command:
            - sh
            - -c
            - |
              ORDINAL=$(hostname | awk -F'-' '{print $NF}')
              echo "$ORDINAL" > /config/node-id
          volumeMounts:
            - name: kafka-config
              mountPath: /config

        - name: format-storage
          image: apache/kafka:4.2.0
          command:
            - sh
            - -c
            - |
              NODE_ID=$(cat /config/node-id)

              if [ ! -f "/data/meta.properties" ]; then
                echo "Formatting storage for node $NODE_ID..."

                echo "node.id=$NODE_ID" > /tmp/kraft.properties
                echo "process.roles=broker,controller" >> /tmp/kraft.properties
                echo "controller.quorum.voters=0@kafka-0.kafka-headless.kafka.svc.cluster.local:9093,1@kafka-1.kafka-headless.kafka.svc.cluster.local:9093,2@kafka-2.kafka-headless.kafka.svc.cluster.local:9093" >> /tmp/kraft.properties
                echo "listeners=PLAINTEXT://:9092,CONTROLLER://:9093" >> /tmp/kraft.properties
                echo "advertised.listeners=PLAINTEXT://localhost:9092" >> /tmp/kraft.properties
                echo "listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT" >> /tmp/kraft.properties
                echo "inter.broker.listener.name=PLAINTEXT" >> /tmp/kraft.properties
                echo "controller.listener.names=CONTROLLER" >> /tmp/kraft.properties
                echo "log.dirs=/data" >> /tmp/kraft.properties

                /opt/kafka/bin/kafka-storage.sh format \
                  --ignore-formatted \
                  --cluster-id q1Sh-9_ISia_zwGINzRvyQ \
                  --config /tmp/kraft.properties
              else
                echo "Already formatted, skipping."
              fi
          volumeMounts:
            - name: kafka-data
              mountPath: /data
            - name: kafka-config
              mountPath: /config

      containers:
        - name: kafka
          image: apache/kafka:4.2.0

          command:
            - sh
            - -c
            - |
              export KAFKA_NODE_ID=$(cat /config/node-id)
              exec /etc/kafka/docker/run

          ports:
            - containerPort: 9092
            - containerPort: 9093

          env:
            - name: CLUSTER_ID
              value: "q1Sh-9_ISia_zwGINzRvyQ"

            - name: KAFKA_PROCESS_ROLES
              value: "broker,controller"

            - name: KAFKA_CONTROLLER_LISTENER_NAMES
              value: "CONTROLLER"

            - name: KAFKA_CONTROLLER_QUORUM_VOTERS
              value: "0@kafka-0.kafka-headless.kafka.svc.cluster.local:9093,1@kafka-1.kafka-headless.kafka.svc.cluster.local:9093,2@kafka-2.kafka-headless.kafka.svc.cluster.local:9093"

            - name: KAFKA_LISTENERS
              value: "INTERNAL://:9092,CONTROLLER://:9093"

            - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
              value: "INTERNAL:SASL_PLAINTEXT,CONTROLLER:PLAINTEXT"

            - name: KAFKA_INTER_BROKER_LISTENER_NAME
              value: "INTERNAL"

            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name

            - name: KAFKA_ADVERTISED_LISTENERS
              value: "INTERNAL://$(POD_NAME).kafka-headless.kafka.svc.cluster.local:9092"

            - name: KAFKA_SASL_ENABLED_MECHANISMS
              value: SCRAM-SHA-512

            - name: KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL
              value: SCRAM-SHA-512

            - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
              value: "3"

            - name: KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR
              value: "3"

            - name: KAFKA_TRANSACTION_STATE_LOG_MIN_ISR
              value: "2"

            - name: KAFKA_MIN_INSYNC_REPLICAS
              value: "2"

            - name: KAFKA_LOG_DIRS
              value: /data

            - name: KAFKA_OPTS
              value: "-Djava.security.auth.login.config=/opt/kafka/config/jaas/jaas.conf"

          volumeMounts:
            - name: kafka-data
              mountPath: /data
            - name: kafka-config
              mountPath: /config
            - name: kafka-jaas
              mountPath: /opt/kafka/config/jaas

          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: "2"
              memory: 4Gi

  volumeClaimTemplates:
    - metadata:
        name: kafka-data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi

Step 6 — Initial Deployment

Apply all manifests:

kubectl apply -f namespace.yaml
kubectl apply -f kafka-svc.yaml
kubectl apply -f kafka-jaas.yaml
kubectl apply -f kafka-sasl.yaml
kubectl apply -f kafka-stateful-set.yaml

Watch the pods come up:

kubectl get pods -n kafka -w

All three pods should reach Running status within a minute or two. Check kafka-0 logs to confirm a clean startup:

kubectl logs -n kafka kafka-0

A healthy startup ends with:

[KafkaRaftServer nodeId=0] Kafka Server started

Step 7 — Bootstrap SCRAM Credentials

This is the most involved post-deployment step, and also the most commonly misunderstood. Here is why it requires a special process.

The Chicken-and-Egg Problem

The broker's INTERNAL listener (port 9092) requires SASL/SCRAM-SHA-512 authentication. The kafka-configs.sh admin tool needs to connect to a broker to write SCRAM credentials into Kafka's metadata. But to connect to the broker, you need valid SCRAM credentials — which don't exist yet because you haven't written them.

There are two apparent escape hatches that do not actually work:

Using the JAAS file credentials directly against port 9092 — the JAAS file tells the broker process what credentials to use for its own inter-broker connections, but those credentials must also exist in Kafka's metadata store before the SCRAM handshake will succeed. The JAAS file does not pre-populate metadata.
Using --bootstrap-controller against port 9093 — Kafka 4.x does not support the alterUserScramCredentials admin API via the controller quorum endpoint. It must go through a broker.

The solution is to temporarily open an unauthenticated PLAINTEXT listener on a separate port (9094), use it to write the credentials, then close it again.

7a — Add the Temporary Listener to the Service

Edit kafka-svc.yaml to add port 9094:

# kafka-svc.yaml (temporary — port 9094 will be removed after bootstrap)
apiVersion: v1
kind: Service
metadata:
  name: kafka-headless
  namespace: kafka
spec:
  clusterIP: None
  selector:
    app: kafka
  ports:
    - name: internal
      port: 9092
    - name: controller
      port: 9093
    - name: plaintext-bootstrap
      port: 9094

7b — Add the Temporary Listener to the StatefulSet

Edit kafka-stateful-set.yaml and make these changes to the broker container:

Add containerPort: 9094 to the ports list.

Change KAFKA_LISTENERS to:

- name: KAFKA_LISTENERS
  value: "INTERNAL://:9092,CONTROLLER://:9093,PLAINTEXT://:9094"

Change KAFKA_LISTENER_SECURITY_PROTOCOL_MAP to:

- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
  value: "INTERNAL:SASL_PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT"

Change KAFKA_ADVERTISED_LISTENERS to:

- name: KAFKA_ADVERTISED_LISTENERS
  value: "INTERNAL://$(POD_NAME).kafka-headless.kafka.svc.cluster.local:9092,PLAINTEXT://$(POD_NAME).kafka-headless.kafka.svc.cluster.local:9094"

7c — Apply and Wait for Rollout

kubectl apply -f kafka-svc.yaml
kubectl apply -f kafka-stateful-set.yaml
kubectl rollout status statefulset/kafka -n kafka

7d — Register the SCRAM Credentials

kubectl exec -n kafka kafka-0 -- \
  /opt/kafka/bin/kafka-configs.sh \
  --bootstrap-server kafka-0.kafka-headless.kafka.svc.cluster.local:9094 \
  --alter \
  --add-config 'SCRAM-SHA-512=[password=supersecret]' \
  --entity-type users \
  --entity-name admin

Expected output:

Completed updating config for user admin.

If you need additional users (application service accounts), register them now while port 9094 is still open:

kubectl exec -n kafka kafka-0 -- \
  /opt/kafka/bin/kafka-configs.sh \
  --bootstrap-server kafka-0.kafka-headless.kafka.svc.cluster.local:9094 \
  --alter \
  --add-config 'SCRAM-SHA-512=[password=apppassword]' \
  --entity-type users \
  --entity-name myapp

7e — Remove the Temporary Listener

Revert kafka-svc.yaml to remove port 9094:

# kafka-svc.yaml (final)
apiVersion: v1
kind: Service
metadata:
  name: kafka-headless
  namespace: kafka
spec:
  clusterIP: None
  selector:
    app: kafka
  ports:
    - name: internal
      port: 9092
    - name: controller
      port: 9093

Revert the StatefulSet changes — remove the PLAINTEXT entries from KAFKA_LISTENERS, KAFKA_LISTENER_SECURITY_PROTOCOL_MAP, and KAFKA_ADVERTISED_LISTENERS, and remove the containerPort: 9094 line.

Apply and wait:

kubectl apply -f kafka-svc.yaml
kubectl apply -f kafka-stateful-set.yaml
kubectl rollout status statefulset/kafka -n kafka

Step 8 — Verify the Cluster

Check the KRaft quorum status:

kubectl exec -n kafka kafka-0 -- \
  /opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-controller kafka-0.kafka-headless.kafka.svc.cluster.local:9093 \
  describe --status

A healthy cluster shows:

LeaderId:             0
LeaderEpoch:          1
HighWatermark:        303
MaxFollowerLag:       0
MaxFollowerLagTimeMs: 0
CurrentVoters:        [{"id": 0, ...}, {"id": 1, ...}, {"id": 2, ...}]
CurrentObservers:     []

MaxFollowerLag: 0 confirms all three nodes are fully in sync. Three voters and no observers confirms all nodes are full participants in the quorum.

Verify SASL authentication works on port 9092:

kubectl exec -n kafka kafka-0 -- bash -c '
cat > /tmp/client.properties << EOF
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="supersecret";
EOF
/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka-0.kafka-headless.kafka.svc.cluster.local:9092 \
  --command-config /tmp/client.properties \
  --list
'

Create a test topic and produce/consume a message:

kubectl exec -n kafka kafka-0 -- bash -c '
cat > /tmp/client.properties << EOF
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="supersecret";
EOF

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka-0.kafka-headless.kafka.svc.cluster.local:9092 \
  --command-config /tmp/client.properties \
  --create --topic test --partitions 3 --replication-factor 3

echo "hello kafka" | /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server kafka-0.kafka-headless.kafka.svc.cluster.local:9092 \
  --producer.config /tmp/client.properties \
  --topic test

/opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server kafka-0.kafka-headless.kafka.svc.cluster.local:9092 \
  --consumer.config /tmp/client.properties \
  --topic test --from-beginning --max-messages 1
'

Production Hardening Checklist

The cluster at this point is functional and secured with SASL. The following additional steps are needed before handling real production traffic.

Add TLS Encryption

The current setup uses SASL_PLAINTEXT, meaning credentials and data are transmitted unencrypted within the cluster. For any environment where network traffic could be intercepted, upgrade to SASL_SSL.

Generate a keystore and truststore using keytool. The Subject Alternative Names (SANs) must cover all broker hostnames — this is the most common mistake when setting up TLS for Kafka on Kubernetes.

PASS=yourpassword
SAN="dns:kafka-0.kafka-headless.kafka.svc.cluster.local,\
dns:kafka-1.kafka-headless.kafka.svc.cluster.local,\
dns:kafka-2.kafka-headless.kafka.svc.cluster.local,\
dns:kafka-headless.kafka.svc.cluster.local"

# Generate CA
keytool -genkeypair -alias ca -keyalg RSA -keysize 2048 \
  -dname "CN=kafka-ca" -validity 3650 \
  -keystore ca.jks -storepass $PASS -storetype JKS

keytool -exportcert -alias ca -keystore ca.jks \
  -storepass $PASS -file ca.crt

# Generate broker keypair and CSR
keytool -genkeypair -alias kafka -keyalg RSA -keysize 2048 \
  -dname "CN=kafka" -validity 3650 \
  -keystore keystore.jks -storepass $PASS -storetype JKS

keytool -certreq -alias kafka -keystore keystore.jks \
  -storepass $PASS -file kafka.csr

# Sign the CSR with the CA, including all broker SANs
keytool -gencert -alias ca -keystore ca.jks -storepass $PASS \
  -infile kafka.csr -outfile kafka.crt -validity 3650 \
  -ext "SAN=$SAN"

# Import CA + signed cert into the keystore
keytool -importcert -alias ca -file ca.crt \
  -keystore keystore.jks -storepass $PASS -noprompt
keytool -importcert -alias kafka -file kafka.crt \
  -keystore keystore.jks -storepass $PASS -noprompt

# Build truststore containing only the CA
keytool -importcert -alias ca -file ca.crt \
  -keystore truststore.jks -storepass $PASS -noprompt

# Store in Kubernetes
kubectl create secret generic kafka-tls -n kafka \
  --from-file=keystore.jks=keystore.jks \
  --from-file=truststore.jks=truststore.jks \
  --from-literal=password=$PASS \
  --dry-run=client -o yaml | kubectl apply -f -

Then update the StatefulSet to mount the secret and change the listener protocols:

INTERNAL:SASL_PLAINTEXT → INTERNAL:SASL_SSL
CONTROLLER:PLAINTEXT → CONTROLLER:SSL
Add KAFKA_SSL_KEYSTORE_LOCATION, KAFKA_SSL_TRUSTSTORE_LOCATION, KAFKA_SSL_KEYSTORE_PASSWORD, KAFKA_SSL_TRUSTSTORE_PASSWORD env vars
Mount the kafka-tls secret as a volume at /tls

Change Default Credentials

The admin/supersecret credentials used in this guide are for demonstration only. Before going to production:

Update kafka-jaas.yaml with a strong randomly generated password
Update kafka-sasl.yaml with the same password
Re-register the SCRAM credentials using the bootstrap process in Step 7
Rotate any client configurations that reference the old credentials

Network Policies

Add a NetworkPolicy to restrict which pods can reach Kafka:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: kafka-network-policy
  namespace: kafka
spec:
  podSelector:
    matchLabels:
      app: kafka
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: kafka
      ports:
        - port: 9092
        - port: 9093
    - from:
        - namespaceSelector: {}
      ports:
        - port: 9092
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: kafka

Pod Anti-Affinity

Prevent Kubernetes from scheduling multiple Kafka pods on the same node:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
            - key: app
              operator: In
              values:
                - kafka
        topologyKey: kubernetes.io/hostname

Add this under spec.template.spec in the StatefulSet.

Pod Disruption Budget

Ensure Kubernetes never takes down more than one Kafka pod at a time during voluntary disruptions such as node drains or rolling updates:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: kafka-pdb
  namespace: kafka
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: kafka

Liveness and Readiness Probes

Add probes so Kubernetes can detect and restart unhealthy pods:

livenessProbe:
  tcpSocket:
    port: 9092
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 6

readinessProbe:
  tcpSocket:
    port: 9092
  initialDelaySeconds: 20
  periodSeconds: 5
  failureThreshold: 3

Resource Tuning

Workload	CPU Request	Memory Request	Storage
Development	250m	512Mi	5Gi
Low traffic	500m	1Gi	20Gi
Medium traffic	1	2Gi	50Gi
High traffic	2–4	4–8Gi	100Gi+

JVM Heap Tuning

Set the JVM heap via the KAFKA_HEAP_OPTS env var. A common rule of thumb is 50% of the container memory limit, not exceeding 6Gi:

- name: KAFKA_HEAP_OPTS
  value: "-Xms2g -Xmx2g"

Monitoring

The official apache/kafka image exposes JMX metrics. Enable them with:

- name: KAFKA_JMX_PORT
  value: "9999"
- name: KAFKA_JMX_HOSTNAME
  value: "0.0.0.0"

Key metrics to monitor:

kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions — should always be 0
kafka.controller:type=KafkaController,name=ActiveControllerCount — should be 1 across the cluster
kafka.network:type=RequestMetrics,name=TotalTimeMs — producer/consumer latency
JVM GC pause times — long pauses indicate heap needs tuning

Common Troubleshooting

Pod stuck in Init state

Check the init container logs:

kubectl logs -n kafka kafka-0 -c format-storage
kubectl logs -n kafka kafka-0 -c init-node-id

No readable meta.properties error

The broker cannot find formatted storage. Ensure you are using /etc/kafka/docker/run as the entrypoint, not kafka-server-start.sh. The latter ignores KAFKA_* env vars and the broker looks in the wrong directory.

Invalid cluster.id error

The CLUSTER_ID env var does not match what was written to meta.properties during the format step. Delete the PVCs to force a reformat:

kubectl delete pvc -n kafka -l app=kafka

There must be at least one broker advertised listener during format

The temp kraft.properties used in the format-storage init container must include a non-controller listener. Ensure listeners, advertised.listeners, listener.security.protocol.map, and inter.broker.listener.name are all present in the temp properties file.

SASL authentication timeout on port 9092

The SCRAM credentials have not been registered. The JAAS file alone is not enough — run the bootstrap process in Step 7 to write the credentials into Kafka's metadata store.

UnsupportedEndpointTypeException when using --bootstrap-controller

The alterUserScramCredentials admin API is not supported via the controller quorum endpoint in Kafka 4.x. You must connect through a broker (port 9092). Use the temporary PLAINTEXT listener on port 9094 as described in Step 7.

Port 9094 connection refused

The headless service does not expose port 9094 by default. You must explicitly add it to kafka-svc.yaml during the bootstrap process (Step 7a) and remove it afterwards.

Brokers cannot reach each other

Verify the headless service exists and pod DNS resolves from within a pod:

kubectl exec -n kafka kafka-0 -- \
  nslookup kafka-1.kafka-headless.kafka.svc.cluster.local

MaxFollowerLag is non-zero

One broker is behind. Check its logs for errors. Common causes are disk I/O pressure, a recent pod restart, or network instability.

Upgrading Kafka

To upgrade to a new Kafka version, update the image field in the StatefulSet. Kubernetes performs a rolling restart one pod at a time. Since KRaft requires a majority, the cluster stays available as long as at least 2 of 3 pods are running.

kubectl set image statefulset/kafka \
  kafka=apache/kafka:4.3.0 \
  -n kafka

kubectl rollout status statefulset/kafka -n kafka

Summary

Step	What it does
namespace.yaml	Isolates all Kafka resources
kafka-svc.yaml	Headless service for stable pod DNS
kafka-jaas.yaml	JAAS config for inter-broker SCRAM auth
kafka-sasl.yaml	Secret for client applications
kafka-stateful-set.yaml	Brokers, init containers, storage, config
Step 7 — SCRAM bootstrap	Temporarily open port 9094, write credentials to metadata, close port 9094

The cluster is production-ready when TLS is enabled, credentials are rotated to strong values, pod anti-affinity and a PodDisruptionBudget are in place, resource limits are tuned for your workload, and monitoring is active.

How GNOME drives me Crazy on Ubuntu

giveitatry — Fri, 20 Feb 2026 19:22:06 +0000

If you’re using Ubuntu with the default GNOME desktop, you’ve probably experienced this:

You click a pinned app in the dock (for example Firefox), and instead of switching to your already open window… GNOME shows you thumbnails of all open windows.

This makes me crazy. I need additional click to switch the app.

I don’t want previews.
I just want it to switch to the next window.

Luckily, Ubuntu makes this easy to fix.

Why This Happens

Ubuntu uses a modified GNOME dock called Ubuntu Dock, which is based on Dash to Dock.

By default, clicking a pinned app:

Shows window previews if multiple windows are open.

But we can change that behavior.

The Fix: Make Click Cycle Through Windows

Step 1 – Open Dock Settings

Run this command in Terminal:

gnome-extensions prefs ubuntu-dock@ubuntu.com

Step 2 – Change Click Behavior

Go to the Behavior tab.
Find Click action
Change it to:

Cycle through windows

That’s it.

How to Add a Subdomain with Hetzner DNS API

giveitatry — Sun, 01 Feb 2026 18:19:42 +0000

Hetzner’s DNS Public API allows you to programmatically manage your domains, including creating subdomains. A subdomain is simply a DNS record under your main zone.

1. Get Your Zone ID

Each domain in Hetzner DNS has a unique zone_id. To find it:

curl -X GET "https://dns.hetzner.com/api/v1/zones" \
  -H "Auth-API-Token: $API_TOKEN"

Look for your domain in the response and note its id. For example:

{
  "id": "ABC...",
  "name": "domain.com"
}

2. Add the Subdomain Record

Once you have the zone_id, you can create a subdomain by adding a DNS record. For example, to create xxx.domain.com pointing to 1.2.3.4:

curl -X POST "https://dns.hetzner.com/api/v1/records" \
  -H "Auth-API-Token: $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "zone_id": "ABC...",
    "type": "A",
    "name": "xxx",
    "value": "1.2.3.4",
    "ttl": 300
  }'

Parameters:

zone_id → The zone where the subdomain will be added.
type → DNS record type (A, CNAME, MX, etc.).
name → Subdomain prefix (xxx).
value → IP address or target domain (1.2.3.4 here).
ttl → Time-to-live in seconds (optional, default 300).

3. Verify the Subdomain

To confirm the subdomain was created:

curl -X GET "https://dns.hetzner.com/api/v1/records?zone_id=ABC..." \
  -H "Auth-API-Token: $API_TOKEN"

Check for your new record in the response.

✅ Summary

Subdomains are just DNS records in Hetzner.
Use GET /zones to find your zone_id.
Use POST /records to add subdomains.
Use GET /records to list or verify existing subdomains.

This approach lets you automate DNS management for your domains using Hetzner’s DNS API.

How to Clone a GitLab Repository with a Self-Signed Certificate

giveitatry — Sun, 01 Feb 2026 12:55:26 +0000

When working with a GitLab instance using a self-signed SSL certificate, attempting to clone a repository over HTTPS often fails with an error like:

fatal: unable to access 'https://gitlab.example.com/group/project.git/': 
server certificate verification failed. CAfile: none CRLfile: none

This happens because Git does not trust self-signed certificates by default. Below are safe, step-by-step ways to fix it.

1. Understanding the Problem

Git verifies SSL certificates to ensure the server you’re connecting to is legitimate. Self-signed certificates aren’t trusted by default because they’re not signed by a recognized Certificate Authority (CA).

This means HTTPS connections fail unless Git is explicitly told to trust the certificate.

2. Export the Self-Signed Certificate

Option A: Export via Chrome

Open your GitLab URL in Chrome:

https://gitlab.example.com

Click the lock icon in the address bar → Connection is secure → Certificate is valid.
Go to the Details tab → Click Export….
Save the certificate as gitlab-selfsigned.crt in a permanent location:

Windows: C:\certs\gitlab-selfsigned.crt
Linux/macOS: /home/username/certs/gitlab-selfsigned.crt

Option B: Export via Firefox

Firefox doesn’t always allow direct export from the page, but here are two reliable ways:

Method 1: Using Firefox Preferences

Open Firefox and go to:

about:preferences#privacy

Scroll down to Certificates → Click View Certificates.
Go to the Servers tab, find your GitLab domain (gitlab.example.com), select it → Export…
Save as gitlab-selfsigned.crt.

Method 2: Using OpenSSL (cross-platform, recommended)

Open a terminal (Linux, macOS, or Git Bash on Windows) and run:

echo | openssl s_client -connect gitlab.example.com:443 -showcerts 2>/dev/null | openssl x509 -outform PEM > gitlab-selfsigned.crt

This fetches the certificate directly from GitLab.
Saves it as gitlab-selfsigned.crt in the current directory.

3. Configure Git to Trust the Certificate

Once you have the .crt file:

Windows:

git config --global http.sslCAInfo "C:/certs/gitlab-selfsigned.crt"

Linux / macOS:

git config --global http.sslCAInfo "/home/username/certs/gitlab-selfsigned.crt"

Verify:

git config --global --get http.sslCAInfo

4. Clone the Repository

Now, you can securely clone your repository over HTTPS:

git clone https://gitlab.example.com/group/project.git

5. Alternative: Use SSH Instead of HTTPS

SSH avoids all certificate issues:

Generate an SSH key if you don’t have one:

ssh-keygen -t ed25519 -C "your_email@example.com"

Copy the public key (~/.ssh/id_ed25519.pub) and add it to GitLab:
User Settings → SSH Keys → Add Key
Clone via SSH:

git clone git@gitlab.example.com:group/project.git

No SSL certificate issues occur using SSH.

6. Quick & Unsafe Workaround (Testing Only)

You can temporarily disable SSL verification:

git config --global http.sslVerify false

⚠️ Warning: This is insecure. It allows MITM attacks and should never be used permanently.

7. PyCharm Considerations

PyCharm may use a separate Git executable. To ensure it works:

Go to:

File → Settings → Version Control → Git

Check the Path to Git executable.
Click Test.
Make sure this Git has the http.sslCAInfo configuration set (or use SSH instead).

8. Summary

Recommended workflow for self-signed GitLab:

Export the self-signed certificate (Chrome or Firefox/OpenSSL).
Configure Git to trust the certificate.
Clone via HTTPS.

Alternative: Use SSH to avoid certificate issues entirely.

Avoid: disabling SSL verification permanently — it is insecure.

How to Clone a GitLab Repository with a Self-Signed Certificate

giveitatry — Sun, 01 Feb 2026 08:52:44 +0000

When working with a GitLab instance using a self-signed SSL certificate, attempting to clone a repository over HTTPS often fails with an error like:

fatal: unable to access 'https://gitlab.example.com/group/project.git/': 
server certificate verification failed. CAfile: none CRLfile: none

This happens because Git does not trust self-signed certificates by default. Below are safe, step-by-step ways to fix it.

1. Understanding the Problem

This means HTTPS connections fail unless Git is explicitly told to trust the certificate.

2. Export the Self-Signed Certificate

Option A: Export via Chrome

Open your GitLab URL in Chrome:

https://gitlab.example.com

Click the lock icon in the address bar → Connection is secure → Certificate is valid.
Go to the Details tab → Click Export….
Save the certificate as gitlab-selfsigned.crt in a permanent location:

Windows: C:\certs\gitlab-selfsigned.crt
Linux/macOS: /home/username/certs/gitlab-selfsigned.crt

Option B: Export via Firefox

Firefox doesn’t always allow direct export from the page, but here are two reliable ways:

Method 1: Using Firefox Preferences

Open Firefox and go to:

about:preferences#privacy

Scroll down to Certificates → Click View Certificates.
Go to the Servers tab, find your GitLab domain (gitlab.example.com), select it → Export…
Save as gitlab-selfsigned.crt.

Method 2: Using OpenSSL (cross-platform, recommended)

Open a terminal (Linux, macOS, or Git Bash on Windows) and run:

echo | openssl s_client -connect gitlab.example.com:443 -showcerts 2>/dev/null | openssl x509 -outform PEM > gitlab-selfsigned.crt

This fetches the certificate directly from GitLab.
Saves it as gitlab-selfsigned.crt in the current directory.

3. Configure Git to Trust the Certificate

Once you have the .crt file:

Windows:

git config --global http.sslCAInfo "C:/certs/gitlab-selfsigned.crt"

Linux / macOS:

git config --global http.sslCAInfo "/home/username/certs/gitlab-selfsigned.crt"

Verify:

git config --global --get http.sslCAInfo

4. Clone the Repository

Now, you can securely clone your repository over HTTPS:

git clone https://gitlab.example.com/group/project.git

5. Alternative: Use SSH Instead of HTTPS

SSH avoids all certificate issues:

Generate an SSH key if you don’t have one:

ssh-keygen -t ed25519 -C "your_email@example.com"

Copy the public key (~/.ssh/id_ed25519.pub) and add it to GitLab:
User Settings → SSH Keys → Add Key
Clone via SSH:

git clone git@gitlab.example.com:group/project.git

No SSL certificate issues occur using SSH.

6. Quick & Unsafe Workaround (Testing Only)

You can temporarily disable SSL verification:

git config --global http.sslVerify false

⚠️ Warning: This is insecure. It allows MITM attacks and should never be used permanently.

7. PyCharm Considerations

PyCharm may use a separate Git executable. To ensure it works:

Go to:

File → Settings → Version Control → Git

Check the Path to Git executable.
Click Test.
Make sure this Git has the http.sslCAInfo configuration set (or use SSH instead).

8. Summary

Recommended workflow for self-signed GitLab:

Export the self-signed certificate (Chrome or Firefox/OpenSSL).
Configure Git to trust the certificate.
Clone via HTTPS.

Alternative: Use SSH to avoid certificate issues entirely.

Avoid: disabling SSL verification permanently — it is insecure.

DEV Community: giveitatry

Deploying RustFS (Single Node) on k3s

1. Prerequisites

2. Add RustFS Helm repository

3. Create namespace

4. Prepare values.yaml

5. Install RustFS

6. Verify deployment

7. Access RustFS

Option A: Port-forward (recommended)

Option B: Enable ingress (optional)

8. Login

9. Key Configuration Notes

Standalone vs Distributed

Credentials

Storage

10. Troubleshooting

1. Login does not work

2. Pods fail with "not first disk" or quorum errors

3. TCP connection errors between nodes

4. Ingress fails with validation error

5. Ingress not reachable

6. Cannot access service externally

7. Credentials do not change after upgrade

Conclusion

Backup and Restore of a K3s Cluster Using Velero

1. Overview

2. Installing Velero CLI on Linux

2.1 Download

2.2 Install binary system-wide

2.3 Verify installation

3. Preparing Storage Backend

3.1 Create credentials file

4. Installing Velero in K3s

4.1 Installation command

Key parameters explained

4.2 Verify installation

4.3 Validate storage

5. Creating Backups

5.1 Full cluster backup

5.2 Check status

5.3 Detailed inspection

6. Restoring from Backup

7. Automating Backups

8. Persistent Volume Backups (Important)

Options:

Option A — CSI snapshots (preferred)

Option B — Node-agent (Restic)

9. Troubleshooting

9.1 CLI works but server not found

9.2 Backup fails with FailedValidation

9.3 MissingRegion error

9.4 Storage unavailable

9.5 Connection timeout

9.6 Incorrect protocol (HTTP vs HTTPS)

9.7 Plugin process exited (not an error)

9.8 Inconsistent logs (0/0/1 state)

10. Recovery Strategy for K3s

11. Best Practices

Restarting a Kubernetes Deployment from GitLab CI

Introduction

Kubernetes-side requirements

Kubernetes setup using YAML (recommended)

Full manifest

Creating a non-expiring token (recommended for CI/CD)

Apply configuration

Retrieve the token

Kubernetes setup using CLI (alternative)

Temporary token (not recommended for CI)

Obtaining GitLab environment variables

1. KUBE_TOKEN_DEV

2. KUBE_API_URL

3. KUBE_CA_PEM

GitLab CI configuration

.gitlab-ci.yml

Explanation of the pipeline

Troubleshooting

RoleBinding error: cannot change roleRef

Forbidden errors

Token expired

9.2 Backup fails with `FailedValidation`

1. `KUBE_TOKEN_DEV`

2. `KUBE_API_URL`

3. `KUBE_CA_PEM`

`.gitlab-ci.yml`