DEV Community: giveitatry

Upgrading K3s: Complete Guide

giveitatry — Wed, 01 Apr 2026 06:13:00 +0000

TL;DR

# Check current version
k3s --version

# Upgrade server node to latest stable (run on server first)
curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=stable sh -

# Upgrade to a specific version (upgrade One Minor Version at a Time)
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.32.5+k3s1 sh -

# Then upgrade each worker node (upgrade One Minor Version at a Time)
curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=stable K3S_URL=https://<server-ip>:6443 K3S_TOKEN=<token> sh -

# Verify
kubectl get nodes

Your Current Situation

You're running:

k3s version v1.28.4+k3s2 (6ba6c1b6)
go version go1.20.11

v1.28 reached end-of-life in late 2024. The current stable lines as of April 2026 are v1.32 and v1.33. That means you're roughly 4 minor versions behind, which is important — you cannot jump straight from v1.28 to v1.33 in one shot.

The Golden Rule: One Minor Version at a Time

When attempting to upgrade to a new version of K3s, the Kubernetes version skew policy applies. Ensure that your plan does not skip intermediate minor versions when upgrading.

This means your upgrade path from v1.28 must go:

v1.28 → v1.29 → v1.30 → v1.31 → v1.32 → v1.33

You cannot skip from v1.28 directly to v1.33. Each hop must be done separately, with the cluster verified healthy between each step.

Also: when upgrading, upgrade server nodes first one at a time, then any agent nodes. Never upgrade workers before the server.

Important: Traefik Version Change at v1.32

K3s versions v1.31 and earlier will install Traefik v2, while K3s versions v1.32 and later will install Traefik v3. If you're using Traefik as your ingress controller (which is the K3s default), the jump from v1.31 to v1.32 will upgrade Traefik from v2 to v3. Review the Traefik v3 migration guide before doing that hop if you have custom Traefik configuration or IngressRoute resources.

Important: etcd Warning for v1.34+

If you plan to go all the way to v1.34 or v1.35, there is a critical etcd requirement. The K3s project announced that there is no safe upgrade path from etcd 3.5 to 3.6 (included in v1.34+) unless you are running etcd v3.5.26 first. The January 2025 releases of K3s v1.32 include etcd v3.5.26, so upgrading to at least v1.32 before jumping to v1.34+ is mandatory to avoid cluster data issues.

Method 1: Manual Upgrade via Install Script (Recommended)

You can upgrade K3s by using the installation script, or by manually installing the binary of the desired version. The install script is the easiest approach.

Step 1 — Upgrade the server node, one minor version at a time

# SSH into your server node
ssh root@<server-ip>

# Hop 1: v1.28 → v1.29
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.29.13+k3s1 sh -

# Verify server is healthy before proceeding
kubectl get nodes
k3s --version

# Hop 2: v1.29 → v1.30
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.30.13+k3s1 sh -
kubectl get nodes

# Hop 3: v1.30 → v1.31
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.31.9+k3s1 sh -
kubectl get nodes

# Hop 4: v1.31 → v1.32 (Traefik v2 → v3 happens here!)
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.32.5+k3s1 sh -
kubectl get nodes

# Hop 5: v1.32 → v1.33 (optional, if you want latest stable)
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.33.1+k3s1 sh -
kubectl get nodes

Step 2 — Upgrade each worker node

After each server hop is confirmed healthy, upgrade the workers for that version before proceeding to the next hop:

# SSH into each worker node
ssh root@<worker-ip>

# Run the same version as the server just upgraded to
curl -sfL https://get.k3s.io | \
  INSTALL_K3S_VERSION=v1.29.13+k3s1 \
  K3S_URL=https://<server-ip>:6443 \
  K3S_TOKEN=<your-token> \
  sh -

# Verify worker rejoined
kubectl get nodes

Your token is in /var/lib/rancher/k3s/server/node-token on the server node:

cat /var/lib/rancher/k3s/server/node-token

Step 3 — Verify after each hop

# All nodes should show same version and Ready status
kubectl get nodes -o wide

# Check nothing is broken
kubectl get pods -A | grep -v Running | grep -v Completed

Method 2: Upgrade via Binary (Manual)

If you prefer more control, or the install script fails:

# Download the specific version binary
wget https://github.com/k3s-io/k3s/releases/download/v1.29.13%2Bk3s1/k3s

# Replace the binary
chmod +x k3s
sudo mv k3s /usr/local/bin/k3s

# Restart the service
sudo systemctl restart k3s        # on server
sudo systemctl restart k3s-agent  # on workers

# Verify
k3s --version

Draining Nodes Before Upgrading (For Production)

Containers for pods continue running even when K3s is stopped. If your workload is sensitive to brief API server outages, you should manually drain and cordon the node before restarting K3s, and uncordon it afterwards.

For a production cluster where you can't afford any disruption:

# Before upgrading a node, drain it from the server
kubectl drain <node-name> \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --force

# Run the upgrade on that node
# ...

# After the node is back up, uncordon it
kubectl uncordon <node-name>

After the Full Upgrade

Once all nodes are on the new version, do a full health check:

# All nodes Ready and on same version
kubectl get nodes -o wide

# No pods stuck in bad state
kubectl get pods -A | grep -v Running | grep -v Completed

# Check system pods specifically
kubectl get pods -n kube-system

# Verify cert status (especially useful since you're now in cert rotation territory)
sudo k3s certificate check

Summary

Your version	Target	Hops required
v1.28.4	v1.29	1
v1.28.4	v1.32	4 (go through 29, 30, 31, 32)
v1.28.4	v1.33	5 (go through 29, 30, 31, 32, 33)

Take it one minor version at a time, always server first then workers, verify healthy between each hop, and mind the Traefik v2→v3 change when crossing v1.32.

K3s Certificate Rotation: Complete Guide to Managing and Rotating Certificates

giveitatry — Wed, 01 Apr 2026 06:07:30 +0000

TL;DR — All Commands You Need

# 1. Check certificate status on any node
sudo k3s certificate check

# 2. Rotate on the SERVER node
sudo systemctl stop k3s
sudo k3s certificate rotate
sudo systemctl start k3s

# 3. Update kubeconfig after server rotation
sudo cp /etc/rancher/k3s/k3s.yaml $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# 4. IMMEDIATELY restart k3s-agent on EVERY worker node
#    (workers will disconnect after server cert rotation — this fixes it)
ssh root@<worker-ip>
sudo systemctl restart k3s-agent

# 5. Verify all nodes are back
kubectl get nodes -w

Certificate Types and Their Lifetimes

K3s manages two fundamentally different categories of certificates, and confusing them is the most common source of operational mistakes.

Client and Server Certificates — 365 days

K3s client and server certificates are valid for 365 days from their date of issuance. These are the leaf certificates used by every component in the cluster for mutual TLS: the API server, kubelet, etcd, kube-proxy, controller-manager, scheduler, and others. They are signed by the cluster's CA certificates.

The full list of rotatable services is: admin, api-server, controller-manager, scheduler, k3s-controller, k3s-server, cloud-controller, etcd, auth-proxy, kubelet, kube-proxy.

CA (Certificate Authority) Certificates — 10 years

By default, K3s generates self-signed CA certificates during startup of the first server node. These CA certificates are valid for 10 years from date of issuance, and are not automatically renewed. They live in /var/lib/rancher/k3s/server/tls and are the root of trust for the entire cluster. The authoritative copy is stored encrypted in the datastore (etcd or SQLite), with copies extracted to disk on startup.

Automatic Certificate Renewal — How It Works

Any certificates that are expired or within 120 days of expiring are automatically renewed every time K3s starts. This renewal reuses the existing keys, and extends the lifetime of the existing certificates.

This is the key distinction: automatic renewal extends the same keys, it doesn't rotate them. If you want brand new keys generated — for example after a security incident — you need the rotate subcommand covered below.

Version note: Prior to the May 2025 releases (v1.33.1+k3s1, v1.32.5+k3s1, v1.31.9+k3s1, v1.30.13+k3s1), alerts and rotation were triggered at 90 days instead of 120 days. If you're running an older cluster, the threshold is 90 days.

The Practical Implication

If your cluster nodes are rebooted or K3s is restarted at least once every few months as part of normal patching, certificates will silently renew themselves without any operator action. It is expected that you would be taking your hosts down periodically for patching and upgrading every few months — but reality has shown that many do not patch or reboot for more than 3 months, so the best practice is monitoring the certificate expiration.

A cluster that runs for an entire year without a restart will let its certificates expire.

Warning Signs — What You See Before Expiry

When a certificate is within 120 days of expiring, a Kubernetes Warning Event with reason: CertificateExpirationWarning is created, with a relation to the Node using the certificate.

You'll see events like this in your cluster logs:

Warning  CertificateExpirationWarning  Node certificates require attention -
restart k3s on this node to trigger automatic rotation:
kube-proxy/client-kube-proxy.crt: certificate CN=system:kube-proxy will
expire within 90 days at 2025-05-03T12:14:51Z

On newer versions these appear via:

kubectl get events -A --field-selector reason=CertificateExpirationWarning

What Happens When Certificates Actually Expire

This is where things get painful. You get something like this from systemctl status k3s:

x509: certificate has expired or is not yet valid

And any kubectl call returns:

Unable to connect to the server: x509: certificate has expired or is not yet valid

What still works: Apps that are currently running continue to run with no issues — workloads don't immediately crash. The data plane keeps operating. What breaks is the control plane: you lose kubectl access entirely, API server communication fails, and no new deployments, scaling, or config changes are possible. If nodes restart or pods crash, Kubernetes cannot reschedule them because the control plane is inaccessible.

The good news: certificates that are already expired are still automatically renewed every time K3s starts — so the recovery procedure is identical to the proactive rotation procedure. K3s is designed to self-heal on restart even in the fully expired state.

Checking Certificate Status

sudo k3s certificate check

Note: the --output table flag was added in the January 2025 releases (v1.32.0+k3s1, v1.31.5+k3s1, v1.30.9+k3s1) and will error on anything older. Check your version first:

k3s --version

# Only use --output table if you're on a January 2025+ release
sudo k3s certificate check --output table

Example output:

FILENAME                    SUBJECT                     USAGES       EXPIRES                  RESIDUAL TIME   STATUS
--------                    -------                     ------       -------                  -------------   ------
client-kube-proxy.crt       system:kube-proxy           ClientAuth   Jun 09, 2026 10:17 UTC   1 year          OK
client-kubelet.crt          system:node:k3s-server-1    ClientAuth   Jun 09, 2026 10:17 UTC   1 year          OK
serving-kubelet.crt         k3s-server-1                ServerAuth   Jun 09, 2026 10:17 UTC   1 year          OK
client-k3s-controller.crt   system:k3s-controller       ClientAuth   Jun 09, 2026 10:17 UTC   1 year          OK

Each certificate file in the FILENAME column contains at least two certificates — the leaf (or end entity) client/server certificate, any intermediate Certificate Authority certificates, and the root Certificate Authority certificate. That's why you see each filename appear twice — the leaf cert at 1 year, and the CA cert at 10 years.

To inspect a specific cert directly with openssl:

sudo openssl x509 \
  -in /var/lib/rancher/k3s/server/tls/client-admin.crt \
  -noout -dates

To inspect the certificate currently embedded in your kubeconfig:

sudo cat /etc/rancher/k3s/k3s.yaml \
  | grep 'client-certificate-data' \
  | awk '{print $2}' \
  | base64 -d \
  | openssl x509 -noout -dates

This is useful to confirm your local kubectl context is using up-to-date credentials, separate from what the node itself reports.

Manual Rotation — Step by Step

On the Server Node

# Step 1: Stop K3s
sudo systemctl stop k3s

# Step 2: Rotate certificates (generates new certs AND new keys)
sudo k3s certificate rotate

# Step 3: Start K3s
sudo systemctl start k3s

After K3s starts back up, the new certificate is written to /etc/rancher/k3s/k3s.yaml. This file is your kubeconfig and now contains the updated client certificate, client key, and certificate authority data:

cd /etc/rancher/k3s
ls
# k3s.yaml

cat k3s.yaml

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <base64-encoded-new-CA>
    server: https://127.0.0.1:6443
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
users:
- name: default
  user:
    client-certificate-data: <base64-encoded-new-cert>
    client-key-data: <base64-encoded-new-key>

# Step 4: Copy the updated kubeconfig
sudo cp /etc/rancher/k3s/k3s.yaml $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# If managing the cluster from a remote machine, copy it there too
scp root@<server-ip>:/etc/rancher/k3s/k3s.yaml ~/.kube/config
# Fix the server address if it points to 127.0.0.1
sed -i 's/127.0.0.1/<your-server-ip>/g' ~/.kube/config

# Step 5: Verify server is healthy
sudo k3s certificate check
kubectl get nodes

Any CI/CD pipelines, Helm deployments, or developer workstations that had a copy of the old kubeconfig will also need this updated file — the old client certificate is invalid the moment rotation runs.

Rotating Specific Services Only

You can limit rotation to specific components rather than rotating everything at once:

sudo k3s certificate rotate --service api-server,kubelet
sudo k3s certificate rotate --service etcd
sudo k3s certificate rotate --service admin,controller-manager,scheduler

This is useful when only certain certs are approaching expiry, or when you want to minimize the blast radius in a sensitive environment. The stop/start of K3s is still required around this command.

⚠️ Warning: Worker Nodes Will Disconnect After Rotation

This is something the official documentation doesn't warn you about clearly enough, and it can cause a cascading failure that looks far worse than a simple cert issue.

What Happens

When you rotate certificates on the server node, the API server starts presenting new certificates. Worker nodes running k3s-agent were trusting the old certificates. Until the agent is restarted and picks up the new trust chain, kubelet on each worker loses the ability to communicate with the API server:

Kubelet stopped posting node status.

The workers show as NotReady even though the machines themselves are perfectly healthy.

The Cascade With Local Storage

If you're using local-path storage (the K3s default), this gets significantly worse:

You rotate certs on the server 🔄
Kubelet on workers stops trusting the API server ❌
Workers become NotReady ❌
PVCs using local-path are physically locked to those dead nodes ❌
Pods tied to that storage cannot reschedule ❌
Scheduler starts throwing volume node affinity conflict ❌
Any stateful workload (databases, etc.) is now stuck in Pending or Initializing forever ❌

The fix is straightforward — but you have to do it immediately after rotating the server, before anything starts crashing and trying to reschedule.

Fix — Restart the Agent on Every Worker Node

SSH into each worker and restart the agent:

# Step 1 — SSH into the worker
ssh root@<worker-ip>

# Step 2 — Restart k3s-agent
sudo systemctl restart k3s-agent

# Step 3 — Check status
sudo systemctl status k3s-agent

If the agent still won't come back after a restart:

# Full reboot as last resort
sudo reboot

Then verify from the control plane that all nodes are back:

kubectl get nodes -w

Wait for all workers to flip back to Ready. Once they do, any stuck pods will reschedule automatically — no manual intervention on the workloads themselves is needed.

The Correct Order of Operations for a Full Cluster Rotation

To avoid the disconnection cascade entirely, always treat cert rotation as a coordinated cluster-wide operation:

1. Stop k3s on server           →  sudo systemctl stop k3s
2. Rotate certs on server       →  sudo k3s certificate rotate
3. Start k3s on server          →  sudo systemctl start k3s
4. Update kubeconfig            →  cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
5. Restart agent on each worker →  sudo systemctl restart k3s-agent
6. Verify all nodes Ready       →  kubectl get nodes

Don't wait between steps 4 and 5. The longer workers run without restarting their agent, the higher the chance that a pod crash or node pressure event triggers a rescheduling attempt that hits the volume node affinity conflict wall.

CA Certificate Rotation

CA rotation is a completely different operation from leaf cert rotation. To rotate CA certificates and keys, use the k3s certificate rotate-ca command. The command performs integrity checks to confirm that the updated certificates and keys are usable. If the updated data is acceptable, the datastore's encrypted bootstrap key is updated, and the new certificates and keys will be used the next time K3s starts. If problems are encountered while validating the certificates and keys, an error is reported to the system log and the operation is cancelled without changes.

Non-disruptive vs. Disruptive Rotation

A cluster that has been started with custom CA certificates can renew or rotate the CA certificates and keys non-disruptively, as long as the same root CA is used. If a new root CA is required, the rotation will be disruptive. The k3s certificate rotate-ca --force option must be used, all nodes that were joined with a secure token (including servers) will need to be reconfigured to use the new token value, and pods will need to be restarted to trust the new root CA.

The safe path is to generate new intermediate CAs signed by the existing root CA — existing trust chains remain valid, nodes can rejoin without token changes, and pods don't need restarts.

CA Rotation Procedure

# Stage new CA certificates into a TEMP directory
# Do NOT place them directly into the live /server/tls directory
mkdir -p /tmp/k3s-ca-rotate

# Use the helper script from the K3s repo to generate new CAs
curl -o /tmp/generate-custom-ca-certs.sh \
  https://raw.githubusercontent.com/k3s-io/k3s/main/contrib/util/generate-custom-ca-certs.sh
chmod +x /tmp/generate-custom-ca-certs.sh
K3S_DATADIR=/tmp/k3s-ca-rotate /tmp/generate-custom-ca-certs.sh

# Load the new CAs into the datastore
sudo k3s certificate rotate-ca --path /tmp/k3s-ca-rotate/server/tls

# Restart K3s on all nodes — servers first, then agents
sudo systemctl restart k3s

Back up the generated root and intermediate CA files somewhere safe after this — you'll need them if you ever need to rotate CAs again in the future.

Monitoring — Preventing Surprise Expiry

Watch for Kubernetes warning events:

kubectl get events -A \
  --field-selector reason=CertificateExpirationWarning \
  --watch

Prometheus alert rule (works with kube-prometheus-stack):

- alert: K3sCertificateExpiringSoon
  expr: |
    (k3s_certificate_expiry_seconds - time()) < 30 * 24 * 3600
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "K3s certificate expiring in less than 30 days on {{ $labels.node }}"

Simple weekly cron check on the server node:

#!/bin/bash
# /etc/cron.weekly/k3s-cert-check
k3s certificate check | mail -s "K3s cert check on $(hostname)" ops@yourcompany.com

Summary Table

Certificate type	Validity	Auto-renewal	Rotation command
Client/Server leaf certs	365 days	Yes, on restart (if < 120 days remaining)	`k3s certificate rotate`
CA certificates	10 years	No	`k3s certificate rotate-ca`
kubeconfig embedded cert	Same as leaf	Follows leaf rotation	Copy updated `k3s.yaml`

Key Takeaways

Restart K3s regularly. Even just for patching every few months — this is what triggers automatic certificate renewal. A cluster that never restarts will expire its certs at the 1-year mark.
rotate ≠ restart. A plain restart extends existing keys. k3s certificate rotate generates entirely new keys. Use rotate when you need cryptographic freshness, not just extended validity.
Always restart worker agents immediately after rotating the server. Workers disconnect the moment the server starts presenting new certs. Don't wait — do it before anything tries to reschedule.
local-path storage amplifies worker disconnection. PVCs are locked to specific nodes. If a worker goes NotReady while holding a PVC, any pod using that storage is stuck until the worker comes back.
Update your kubeconfig after server node rotation — the embedded client certificate changes, and any CI/CD pipelines using the old kubeconfig will break immediately.
CA certs are your 10-year bomb. Mark a calendar reminder. They won't warn you and won't auto-renew.
Backups are automatic. Old certs are saved to /var/lib/rancher/k3s/server/tls-<timestamp> before rotation, giving you a rollback path.

Running a TeamCity Agent as a systemd Service

giveitatry — Tue, 24 Mar 2026 06:44:39 +0000

Running a TeamCity agent as a system service ensures it continues working after logout and automatically starts on boot. This is the recommended production setup.

📁 Prerequisites

Make sure your TeamCity agent is installed in:

/opt/teamcity/bin/agent.sh

👉 The agent.sh script must exist inside the bin directory:

/opt/teamcity/bin/agent.sh

Also ensure it is executable:

chmod +x /opt/teamcity/bin/agent.sh

⚙️ Create systemd service

Create the file:

sudo nano /etc/systemd/system/teamcity-agent.service

Paste the following configuration:

[Unit]
Description=TeamCity Build Agent
After=network.target

[Service]
Type=oneshot
User=root
Group=root
WorkingDirectory=/opt/teamcity

ExecStart=/opt/teamcity/bin/agent.sh start
ExecStop=/opt/teamcity/bin/agent.sh stop

RemainAfterExit=yes
SuccessExitStatus=0 143

[Install]
WantedBy=multi-user.target

🚀 Enable and start the service

Reload systemd and start the agent:

sudo systemctl daemon-reload
sudo systemctl enable teamcity-agent
sudo systemctl start teamcity-agent

🔍 Verify status

systemctl status teamcity-agent

Expected result:

Active: active (exited)

This is normal for TeamCity agents when using Type=oneshot.

⚠️ Notes

The agent must be located in /opt/teamcity/bin/agent.sh
RemainAfterExit=yes is required so systemd considers the service active
SuccessExitStatus=143 prevents false failure detection
Using root works, but for production it's safer to use a dedicated user

Kafka 4.2.0 on Kubernetes - Complete Setup Guide - Exposed to Internet

giveitatry — Mon, 23 Mar 2026 08:20:43 +0000

3-broker Kafka cluster on k3s with KRaft, SASL/SCRAM, and external access via Traefik.

Before you start — get your Traefik IP:

kubectl get svc traefik -n kube-system

Copy any IP from the EXTERNAL-IP column. Replace every occurrence of
192.168.1.119 in the YAMLs below with your actual IP.

Stage 1 — Bootstrap

Apply all of these files in order. This stage starts the cluster with port 9094
open (no auth) so you can register SCRAM credentials.

1.1 namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: kafka

kubectl apply -f namespace.yaml

1.2 kafka-jaas.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-jaas
  namespace: kafka
data:
  jaas.conf: |
    KafkaServer {
      org.apache.kafka.common.security.scram.ScramLoginModule required
      username="admin"
      password="supersecret";
    };

kubectl apply -f kafka-jaas.yaml

1.3 kafka-sasl.yaml

apiVersion: v1
kind: Secret
metadata:
  name: kafka-sasl
  namespace: kafka
type: Opaque
stringData:
  username: admin
  password: supersecret

kubectl apply -f kafka-sasl.yaml

1.4 traefik-config.yaml

Opens three TCP entrypoints on Traefik — one per broker.

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    ports:
      kafka-0:
        port: 9092
        expose:
          default: true
        exposedPort: 9092
        protocol: TCP
      kafka-1:
        port: 9093
        expose:
          default: true
        exposedPort: 9093
        protocol: TCP
      kafka-2:
        port: 9094
        expose:
          default: true
        exposedPort: 9094
        protocol: TCP

kubectl apply -f traefik-config.yaml
kubectl rollout status deployment/traefik -n kube-system

Verify ports appeared:

kubectl get svc traefik -n kube-system
# PORT(S) should include 9092, 9093, 9094 alongside 80 and 443

1.5 kafka-traefik.yaml

Headless service for internal pod DNS, one ClusterIP service per broker
(Traefik v3 routes to port 9095 on each pod), and IngressRouteTCP rules.

apiVersion: v1
kind: Service
metadata:
  name: kafka-headless
  namespace: kafka
spec:
  clusterIP: None
  selector:
    app: kafka
  ports:
    - name: internal
      port: 9092
    - name: controller
      port: 9093
    - name: plaintext-bootstrap
      port: 9094

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-0-external
  namespace: kafka
spec:
  type: ClusterIP
  selector:
    app: kafka
    statefulset.kubernetes.io/pod-name: kafka-0
  ports:
    - name: external
      port: 9092
      targetPort: 9095

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-1-external
  namespace: kafka
spec:
  type: ClusterIP
  selector:
    app: kafka
    statefulset.kubernetes.io/pod-name: kafka-1
  ports:
    - name: external
      port: 9092
      targetPort: 9095

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-2-external
  namespace: kafka
spec:
  type: ClusterIP
  selector:
    app: kafka
    statefulset.kubernetes.io/pod-name: kafka-2
  ports:
    - name: external
      port: 9092
      targetPort: 9095

---
apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
  name: kafka-0-tcp
  namespace: kafka
spec:
  entryPoints:
    - kafka-0
  routes:
    - match: HostSNI(`*`)
      services:
        - name: kafka-0-external
          port: 9092

---
apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
  name: kafka-1-tcp
  namespace: kafka
spec:
  entryPoints:
    - kafka-1
  routes:
    - match: HostSNI(`*`)
      services:
        - name: kafka-1-external
          port: 9092

---
apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
  name: kafka-2-tcp
  namespace: kafka
spec:
  entryPoints:
    - kafka-2
  routes:
    - match: HostSNI(`*`)
      services:
        - name: kafka-2-external
          port: 9092

kubectl apply -f kafka-traefik.yaml

1.6 kafka-stateful-bootstrap.yaml

⚠️ Replace 192.168.1.119 with your Traefik IP before applying.

Includes:

EXTERNAL listener on port 9095 — Traefik routes external traffic here
PLAINTEXT listener on port 9094 — temporary, no auth, used to register SCRAM credentials
init container calculates the external port per broker: kafka-0 → 9092, kafka-1 → 9093, kafka-2 → 9094

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
  namespace: kafka
spec:
  serviceName: kafka-headless
  replicas: 3
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      securityContext:
        fsGroup: 1001
      volumes:
        - name: kafka-config
          emptyDir: {}
        - name: kafka-jaas
          configMap:
            name: kafka-jaas
      initContainers:
        - name: init-node-id
          image: busybox:1.36
          command:
            - sh
            - -c
            - |
              ORDINAL=$(hostname | awk -F'-' '{print $NF}')
              echo "$ORDINAL" > /config/node-id
              EXTERNAL_PORT=$((9092 + ORDINAL))
              echo "$EXTERNAL_PORT" > /config/external-port
          volumeMounts:
            - name: kafka-config
              mountPath: /config
        - name: format-storage
          image: apache/kafka:4.2.0
          command:
            - sh
            - -c
            - |
              NODE_ID=$(cat /config/node-id)
              if [ ! -f "/data/meta.properties" ]; then
                echo "Formatting storage for node $NODE_ID..."
                echo "node.id=$NODE_ID" > /tmp/kraft.properties
                echo "process.roles=broker,controller" >> /tmp/kraft.properties
                echo "controller.quorum.voters=0@kafka-0.kafka-headless.kafka.svc.cluster.local:9093,1@kafka-1.kafka-headless.kafka.svc.cluster.local:9093,2@kafka-2.kafka-headless.kafka.svc.cluster.local:9093" >> /tmp/kraft.properties
                echo "listeners=PLAINTEXT://:9092,CONTROLLER://:9093" >> /tmp/kraft.properties
                echo "advertised.listeners=PLAINTEXT://localhost:9092" >> /tmp/kraft.properties
                echo "listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT" >> /tmp/kraft.properties
                echo "inter.broker.listener.name=PLAINTEXT" >> /tmp/kraft.properties
                echo "controller.listener.names=CONTROLLER" >> /tmp/kraft.properties
                echo "log.dirs=/data" >> /tmp/kraft.properties
                /opt/kafka/bin/kafka-storage.sh format \
                  --ignore-formatted \
                  --cluster-id q1Sh-9_ISia_zwGINzRvyQ \
                  --config /tmp/kraft.properties
              else
                echo "Already formatted, skipping."
              fi
          volumeMounts:
            - name: kafka-data
              mountPath: /data
            - name: kafka-config
              mountPath: /config
      containers:
        - name: kafka
          image: apache/kafka:4.2.0
          command:
            - sh
            - -c
            - |
              export KAFKA_NODE_ID=$(cat /config/node-id)
              export EXTERNAL_PORT=$(cat /config/external-port)
              export KAFKA_ADVERTISED_LISTENERS="INTERNAL://$(POD_NAME).kafka-headless.kafka.svc.cluster.local:9092,EXTERNAL://192.168.1.119:${EXTERNAL_PORT},PLAINTEXT://$(POD_NAME).kafka-headless.kafka.svc.cluster.local:9094"
              exec /etc/kafka/docker/run
          ports:
            - containerPort: 9092
            - containerPort: 9093
            - containerPort: 9094
            - containerPort: 9095
          env:
            - name: CLUSTER_ID
              value: "q1Sh-9_ISia_zwGINzRvyQ"
            - name: KAFKA_PROCESS_ROLES
              value: "broker,controller"
            - name: KAFKA_CONTROLLER_LISTENER_NAMES
              value: "CONTROLLER"
            - name: KAFKA_CONTROLLER_QUORUM_VOTERS
              value: "0@kafka-0.kafka-headless.kafka.svc.cluster.local:9093,1@kafka-1.kafka-headless.kafka.svc.cluster.local:9093,2@kafka-2.kafka-headless.kafka.svc.cluster.local:9093"
            - name: KAFKA_LISTENERS
              value: "INTERNAL://:9092,CONTROLLER://:9093,EXTERNAL://:9095,PLAINTEXT://:9094"
            - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
              value: "INTERNAL:SASL_PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:SASL_PLAINTEXT,PLAINTEXT:PLAINTEXT"
            - name: KAFKA_INTER_BROKER_LISTENER_NAME
              value: "INTERNAL"
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: KAFKA_SASL_ENABLED_MECHANISMS
              value: SCRAM-SHA-512
            - name: KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL
              value: SCRAM-SHA-512
            - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
              value: "3"
            - name: KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR
              value: "3"
            - name: KAFKA_TRANSACTION_STATE_LOG_MIN_ISR
              value: "2"
            - name: KAFKA_MIN_INSYNC_REPLICAS
              value: "2"
            - name: KAFKA_LOG_DIRS
              value: /data
            - name: KAFKA_OPTS
              value: "-Djava.security.auth.login.config=/opt/kafka/config/jaas/jaas.conf"
          volumeMounts:
            - name: kafka-data
              mountPath: /data
            - name: kafka-config
              mountPath: /config
            - name: kafka-jaas
              mountPath: /opt/kafka/config/jaas
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: "2"
              memory: 4Gi
  volumeClaimTemplates:
    - metadata:
        name: kafka-data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi

kubectl apply -f kafka-stateful-bootstrap.yaml
kubectl rollout status statefulset/kafka -n kafka

1.7 Register SCRAM credentials

Port 9094 is open and unauthenticated. Use it to write the admin user into
Kafka's metadata store. This is a one-time operation.

kubectl exec -n kafka kafka-0 -- \
  /opt/kafka/bin/kafka-configs.sh \
  --bootstrap-server kafka-0.kafka-headless.kafka.svc.cluster.local:9094 \
  --alter \
  --add-config 'SCRAM-SHA-512=[password=supersecret]' \
  --entity-type users \
  --entity-name admin

Expected output:

Completed updating config for user admin.

To add more users (application service accounts), repeat the command with
different --entity-name and password values while port 9094 is still open.

Stage 2 — Production (close port 9094)

Port 9094 is now unnecessary and a security risk. Apply the final StatefulSet
to remove it. No other files change.

2.1 kafka-stateful-final.yaml

⚠️ Replace 192.168.1.119 with your Traefik IP before applying.

Identical to the bootstrap version except:

PLAINTEXT://:9094 removed from KAFKA_LISTENERS
PLAINTEXT:PLAINTEXT removed from KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
PLAINTEXT entry removed from KAFKA_ADVERTISED_LISTENERS
containerPort: 9094 removed

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
  namespace: kafka
spec:
  serviceName: kafka-headless
  replicas: 3
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      securityContext:
        fsGroup: 1001
      volumes:
        - name: kafka-config
          emptyDir: {}
        - name: kafka-jaas
          configMap:
            name: kafka-jaas
      initContainers:
        - name: init-node-id
          image: busybox:1.36
          command:
            - sh
            - -c
            - |
              ORDINAL=$(hostname | awk -F'-' '{print $NF}')
              echo "$ORDINAL" > /config/node-id
              EXTERNAL_PORT=$((9092 + ORDINAL))
              echo "$EXTERNAL_PORT" > /config/external-port
          volumeMounts:
            - name: kafka-config
              mountPath: /config
        - name: format-storage
          image: apache/kafka:4.2.0
          command:
            - sh
            - -c
            - |
              NODE_ID=$(cat /config/node-id)
              if [ ! -f "/data/meta.properties" ]; then
                echo "Formatting storage for node $NODE_ID..."
                echo "node.id=$NODE_ID" > /tmp/kraft.properties
                echo "process.roles=broker,controller" >> /tmp/kraft.properties
                echo "controller.quorum.voters=0@kafka-0.kafka-headless.kafka.svc.cluster.local:9093,1@kafka-1.kafka-headless.kafka.svc.cluster.local:9093,2@kafka-2.kafka-headless.kafka.svc.cluster.local:9093" >> /tmp/kraft.properties
                echo "listeners=PLAINTEXT://:9092,CONTROLLER://:9093" >> /tmp/kraft.properties
                echo "advertised.listeners=PLAINTEXT://localhost:9092" >> /tmp/kraft.properties
                echo "listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT" >> /tmp/kraft.properties
                echo "inter.broker.listener.name=PLAINTEXT" >> /tmp/kraft.properties
                echo "controller.listener.names=CONTROLLER" >> /tmp/kraft.properties
                echo "log.dirs=/data" >> /tmp/kraft.properties
                /opt/kafka/bin/kafka-storage.sh format \
                  --ignore-formatted \
                  --cluster-id q1Sh-9_ISia_zwGINzRvyQ \
                  --config /tmp/kraft.properties
              else
                echo "Already formatted, skipping."
              fi
          volumeMounts:
            - name: kafka-data
              mountPath: /data
            - name: kafka-config
              mountPath: /config
      containers:
        - name: kafka
          image: apache/kafka:4.2.0
          command:
            - sh
            - -c
            - |
              export KAFKA_NODE_ID=$(cat /config/node-id)
              export EXTERNAL_PORT=$(cat /config/external-port)
              export KAFKA_ADVERTISED_LISTENERS="INTERNAL://$(POD_NAME).kafka-headless.kafka.svc.cluster.local:9092,EXTERNAL://192.168.1.119:${EXTERNAL_PORT}"
              exec /etc/kafka/docker/run
          ports:
            - containerPort: 9092
            - containerPort: 9093
            - containerPort: 9095
          env:
            - name: CLUSTER_ID
              value: "q1Sh-9_ISia_zwGINzRvyQ"
            - name: KAFKA_PROCESS_ROLES
              value: "broker,controller"
            - name: KAFKA_CONTROLLER_LISTENER_NAMES
              value: "CONTROLLER"
            - name: KAFKA_CONTROLLER_QUORUM_VOTERS
              value: "0@kafka-0.kafka-headless.kafka.svc.cluster.local:9093,1@kafka-1.kafka-headless.kafka.svc.cluster.local:9093,2@kafka-2.kafka-headless.kafka.svc.cluster.local:9093"
            - name: KAFKA_LISTENERS
              value: "INTERNAL://:9092,CONTROLLER://:9093,EXTERNAL://:9095"
            - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
              value: "INTERNAL:SASL_PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:SASL_PLAINTEXT"
            - name: KAFKA_INTER_BROKER_LISTENER_NAME
              value: "INTERNAL"
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: KAFKA_SASL_ENABLED_MECHANISMS
              value: SCRAM-SHA-512
            - name: KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL
              value: SCRAM-SHA-512
            - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
              value: "3"
            - name: KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR
              value: "3"
            - name: KAFKA_TRANSACTION_STATE_LOG_MIN_ISR
              value: "2"
            - name: KAFKA_MIN_INSYNC_REPLICAS
              value: "2"
            - name: KAFKA_LOG_DIRS
              value: /data
            - name: KAFKA_OPTS
              value: "-Djava.security.auth.login.config=/opt/kafka/config/jaas/jaas.conf"
          volumeMounts:
            - name: kafka-data
              mountPath: /data
            - name: kafka-config
              mountPath: /config
            - name: kafka-jaas
              mountPath: /opt/kafka/config/jaas
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: "2"
              memory: 4Gi
  volumeClaimTemplates:
    - metadata:
        name: kafka-data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi

kubectl apply -f kafka-stateful-final.yaml
kubectl rollout status statefulset/kafka -n kafka

2.2 Verify

kubectl exec -n kafka kafka-0 -- \
  /opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-controller kafka-0.kafka-headless.kafka.svc.cluster.local:9093 \
  describe --status

Healthy output:

LeaderId:             0
MaxFollowerLag:       0
MaxFollowerLagTimeMs: 0
CurrentVoters:        [{"id": 0, ...}, {"id": 1, ...}, {"id": 2, ...}]
CurrentObservers:     []

2.3 Python client

pip install kafka-python-ng

from kafka import KafkaProducer, KafkaConsumer, KafkaAdminClient
from kafka.admin import NewTopic
from kafka.errors import TopicAlreadyExistsError
import json, time

# Replace 192.168.1.119 with your Traefik IP
BOOTSTRAP_SERVERS = [
    "192.168.1.119:9092",  # kafka-0
    "192.168.1.119:9093",  # kafka-1
    "192.168.1.119:9094",  # kafka-2
]

SASL_CONFIG = {
    "security_protocol": "SASL_PLAINTEXT",
    "sasl_mechanism": "SCRAM-SHA-512",
    "sasl_plain_username": "admin",
    "sasl_plain_password": "supersecret",
}

TOPIC = "orders"


def create_topic(topic):
    admin = KafkaAdminClient(bootstrap_servers=BOOTSTRAP_SERVERS, **SASL_CONFIG)
    try:
        admin.create_topics([
            NewTopic(name=topic, num_partitions=3, replication_factor=3)
        ])
        print(f"Topic '{topic}' created.")
    except TopicAlreadyExistsError:
        print(f"Topic '{topic}' already exists.")
    finally:
        admin.close()


def produce(topic=TOPIC, count=10):
    producer = KafkaProducer(
        bootstrap_servers=BOOTSTRAP_SERVERS,
        **SASL_CONFIG,
        value_serializer=lambda v: json.dumps(v).encode(),
        key_serializer=lambda k: k.encode() if k else None,
        acks="all",
    )
    for i in range(count):
        meta = producer.send(
            topic,
            key=str(i),
            value={"id": i, "ts": time.time()}
        ).get(timeout=10)
        print(f"  sent [{i}] partition={meta.partition} offset={meta.offset}")
    producer.flush()
    producer.close()


def consume(topic=TOPIC):
    consumer = KafkaConsumer(
        topic,
        bootstrap_servers=BOOTSTRAP_SERVERS,
        **SASL_CONFIG,
        value_deserializer=lambda v: json.loads(v.decode()),
        key_deserializer=lambda k: k.decode() if k else None,
        group_id="my-group",
        auto_offset_reset="earliest",
        consumer_timeout_ms=5000,
    )
    for msg in consumer:
        print(
            f"  recv key={msg.key} "
            f"partition={msg.partition} "
            f"offset={msg.offset} "
            f"value={msg.value}"
        )
    consumer.close()


if __name__ == "__main__":
    create_topic(TOPIC)
    produce()
    consume()

Troubleshooting

Pods not starting

kubectl logs -n kafka kafka-0 -c format-storage
kubectl logs -n kafka kafka-0 -c init-node-id

SCRAM registration times out

Port 9094 is not reachable. Check the headless service includes it:

kubectl get svc kafka-headless -n kafka

Python client connects but gets wrong broker addresses

The Traefik IP in the StatefulSet command block is wrong. Fix it and roll:

kubectl rollout restart statefulset/kafka -n kafka

Traefik ports not appearing

kubectl rollout restart deployment/traefik -n kube-system
kubectl get svc traefik -n kube-system

Deploying Apache Kafka 4.2.0 on Kubernetes with KRaft, SASL, and High Availability

giveitatry — Sun, 22 Mar 2026 12:41:13 +0000

Overview

This guide walks through deploying a production-grade Apache Kafka cluster on Kubernetes using KRaft mode (no ZooKeeper), SASL/SCRAM-SHA-512 authentication, and a 3-node StatefulSet for high availability. It covers every manifest file required, the reasoning behind each configuration decision, the SCRAM credential bootstrap process, common pitfalls encountered in practice, and the steps needed to take the cluster from running to production-ready.

Prerequisites

Tools

kubectl configured against your target cluster
A Kubernetes cluster with at least 3 nodes (one per Kafka pod) with sufficient resources
Persistent volume provisioner available (e.g. local-path, Longhorn, Ceph, AWS EBS)
keytool (part of the JDK) if you plan to add TLS later

Cluster Resources

Each Kafka pod in this guide requests 500m CPU and 1Gi RAM, with limits of 2 CPU and 4Gi RAM. For production you should size these based on your throughput requirements. A minimum of 10Gi persistent storage per broker is configured.

Kubernetes Version

Kubernetes 1.21 or later is recommended. StatefulSets, headless Services, and ConfigMaps used in this guide are stable APIs available in all modern versions.

Architecture

The cluster consists of three pods (kafka-0, kafka-1, kafka-2) each running in combined broker+controller mode using KRaft. This means each pod participates in the Raft quorum for metadata management as well as serving producer and consumer traffic.

┌─────────────────────────────────────────┐
│             Kafka Namespace             │
│                                         │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐ │
│  │ kafka-0 │  │ kafka-1 │  │ kafka-2 │ │
│  │ broker  │  │ broker  │  │ broker  │ │
│  │ + ctrl  │  │ + ctrl  │  │ + ctrl  │ │
│  └────┬────┘  └────┬────┘  └────┬────┘ │
│       │             │             │     │
│  port 9092 (SASL_PLAINTEXT — broker)    │
│  port 9093 (PLAINTEXT — controller)     │
│                                         │
│  ┌─────────────────────────────────┐    │
│  │   kafka-headless (ClusterIP:    │    │
│  │   None) — DNS per pod           │    │
│  └─────────────────────────────────┘    │
└─────────────────────────────────────────┘

KRaft requires a majority quorum (2 out of 3) to elect a leader and commit metadata. The cluster can tolerate the loss of one node.

File Structure

You need five manifest files plus one temporary modification during the SCRAM bootstrap process:

namespace.yaml           — Kafka namespace
kafka-svc.yaml           — Headless service for pod DNS
kafka-jaas.yaml          — JAAS config for inter-broker SCRAM auth
kafka-sasl.yaml          — Secret holding admin credentials for client apps
kafka-stateful-set.yaml  — Brokers, init containers, volumes, config

Step 1 — Namespace

# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: kafka

Apply first so all subsequent resources land in the correct namespace:

kubectl apply -f namespace.yaml

Step 2 — Headless Service

# kafka-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: kafka-headless
  namespace: kafka
spec:
  clusterIP: None
  selector:
    app: kafka
  ports:
    - name: internal
      port: 9092
    - name: controller
      port: 9093

A headless service (clusterIP: None) causes Kubernetes DNS to create per-pod A records:

kafka-0.kafka-headless.kafka.svc.cluster.local
kafka-1.kafka-headless.kafka.svc.cluster.local
kafka-2.kafka-headless.kafka.svc.cluster.local

These stable DNS names are what the KRaft quorum voters list and the advertised listeners are built from. They survive pod restarts and rescheduling because they are tied to the StatefulSet ordinal, not the pod IP.

Step 3 — JAAS Configuration

# kafka-jaas.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-jaas
  namespace: kafka
data:
  jaas.conf: |
    KafkaServer {
      org.apache.kafka.common.security.scram.ScramLoginModule required
      username="admin"
      password="supersecret";
    };

This file configures the Java Authentication and Authorization Service (JAAS) for the Kafka broker process itself. The username and password here are the inter-broker credentials — what each broker uses when authenticating to other brokers on the INTERNAL listener.

Important: JAAS alone does not create the SCRAM user in Kafka's metadata store. That is a separate bootstrap step performed after the cluster is running (see Step 7). The JAAS file tells the broker process what credentials to use, but those credentials must also exist in Kafka's internal metadata before inter-broker authentication will succeed.

Step 4 — SASL Secret

# kafka-sasl.yaml
apiVersion: v1
kind: Secret
metadata:
  name: kafka-sasl
  namespace: kafka
type: Opaque
stringData:
  username: admin
  password: supersecret

This secret can be mounted into client pods or referenced by applications that need to connect to Kafka. It is separate from the JAAS ConfigMap so that client credentials can be managed independently of the broker configuration.

Step 5 — StatefulSet

This is the core manifest. It contains two init containers and one main broker container.

Why Two Init Containers?

init-node-id runs a tiny busybox container to extract the pod ordinal (0, 1, or 2) from the hostname and writes it to a shared emptyDir volume. This is necessary because Kubernetes environment variable substitution does not support string manipulation, so you cannot derive the integer 0 from the pod name kafka-0 using env vars alone.

format-storage runs the actual Kafka image to format the KRaft log directory using kafka-storage.sh. It only runs if /data/meta.properties does not already exist, making it safe on pod restarts. The temporary kraft.properties file used during formatting includes a dummy plaintext broker listener — this is required because Kafka's config validator refuses to proceed if the only listener defined is a controller listener. These values are only used during formatting and are not used by the running broker.

Why `/etc/kafka/docker/run` Instead of `kafka-server-start.sh`?

The official apache/kafka Docker image ships an entrypoint at /etc/kafka/docker/run that translates KAFKA_* environment variables into server.properties entries before starting the broker. Calling kafka-server-start.sh directly bypasses this translation entirely — env vars like KAFKA_LOG_DIRS are ignored and the broker falls back to compiled-in defaults, causing it to look for data in the wrong directory.

Why `CLUSTER_ID` as an Env Var?

If CLUSTER_ID is not set, the official entrypoint generates a new random cluster ID on every pod start and attempts to re-format storage. This fails because the existing meta.properties already has a different ID written by the init container. Always set CLUSTER_ID to match the value used during the format step.

Generating Your Own Cluster ID

The cluster ID q1Sh-9_ISia_zwGINzRvyQ used in this guide was generated with:

/opt/kafka/bin/kafka-storage.sh random-uuid

Generate your own unique ID for each cluster you deploy. Do not reuse IDs across clusters.

# kafka-stateful-set.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
  namespace: kafka
spec:
  serviceName: kafka-headless
  replicas: 3

  selector:
    matchLabels:
      app: kafka

  template:
    metadata:
      labels:
        app: kafka

    spec:
      securityContext:
        fsGroup: 1001

      volumes:
        - name: kafka-config
          emptyDir: {}

        - name: kafka-jaas
          configMap:
            name: kafka-jaas

      initContainers:

        - name: init-node-id
          image: busybox:1.36
          command:
            - sh
            - -c
            - |
              ORDINAL=$(hostname | awk -F'-' '{print $NF}')
              echo "$ORDINAL" > /config/node-id
          volumeMounts:
            - name: kafka-config
              mountPath: /config

        - name: format-storage
          image: apache/kafka:4.2.0
          command:
            - sh
            - -c
            - |
              NODE_ID=$(cat /config/node-id)

              if [ ! -f "/data/meta.properties" ]; then
                echo "Formatting storage for node $NODE_ID..."

                echo "node.id=$NODE_ID" > /tmp/kraft.properties
                echo "process.roles=broker,controller" >> /tmp/kraft.properties
                echo "controller.quorum.voters=0@kafka-0.kafka-headless.kafka.svc.cluster.local:9093,1@kafka-1.kafka-headless.kafka.svc.cluster.local:9093,2@kafka-2.kafka-headless.kafka.svc.cluster.local:9093" >> /tmp/kraft.properties
                echo "listeners=PLAINTEXT://:9092,CONTROLLER://:9093" >> /tmp/kraft.properties
                echo "advertised.listeners=PLAINTEXT://localhost:9092" >> /tmp/kraft.properties
                echo "listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT" >> /tmp/kraft.properties
                echo "inter.broker.listener.name=PLAINTEXT" >> /tmp/kraft.properties
                echo "controller.listener.names=CONTROLLER" >> /tmp/kraft.properties
                echo "log.dirs=/data" >> /tmp/kraft.properties

                /opt/kafka/bin/kafka-storage.sh format \
                  --ignore-formatted \
                  --cluster-id q1Sh-9_ISia_zwGINzRvyQ \
                  --config /tmp/kraft.properties
              else
                echo "Already formatted, skipping."
              fi
          volumeMounts:
            - name: kafka-data
              mountPath: /data
            - name: kafka-config
              mountPath: /config

      containers:
        - name: kafka
          image: apache/kafka:4.2.0

          command:
            - sh
            - -c
            - |
              export KAFKA_NODE_ID=$(cat /config/node-id)
              exec /etc/kafka/docker/run

          ports:
            - containerPort: 9092
            - containerPort: 9093

          env:
            - name: CLUSTER_ID
              value: "q1Sh-9_ISia_zwGINzRvyQ"

            - name: KAFKA_PROCESS_ROLES
              value: "broker,controller"

            - name: KAFKA_CONTROLLER_LISTENER_NAMES
              value: "CONTROLLER"

            - name: KAFKA_CONTROLLER_QUORUM_VOTERS
              value: "0@kafka-0.kafka-headless.kafka.svc.cluster.local:9093,1@kafka-1.kafka-headless.kafka.svc.cluster.local:9093,2@kafka-2.kafka-headless.kafka.svc.cluster.local:9093"

            - name: KAFKA_LISTENERS
              value: "INTERNAL://:9092,CONTROLLER://:9093"

            - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
              value: "INTERNAL:SASL_PLAINTEXT,CONTROLLER:PLAINTEXT"

            - name: KAFKA_INTER_BROKER_LISTENER_NAME
              value: "INTERNAL"

            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name

            - name: KAFKA_ADVERTISED_LISTENERS
              value: "INTERNAL://$(POD_NAME).kafka-headless.kafka.svc.cluster.local:9092"

            - name: KAFKA_SASL_ENABLED_MECHANISMS
              value: SCRAM-SHA-512

            - name: KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL
              value: SCRAM-SHA-512

            - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
              value: "3"

            - name: KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR
              value: "3"

            - name: KAFKA_TRANSACTION_STATE_LOG_MIN_ISR
              value: "2"

            - name: KAFKA_MIN_INSYNC_REPLICAS
              value: "2"

            - name: KAFKA_LOG_DIRS
              value: /data

            - name: KAFKA_OPTS
              value: "-Djava.security.auth.login.config=/opt/kafka/config/jaas/jaas.conf"

          volumeMounts:
            - name: kafka-data
              mountPath: /data
            - name: kafka-config
              mountPath: /config
            - name: kafka-jaas
              mountPath: /opt/kafka/config/jaas

          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: "2"
              memory: 4Gi

  volumeClaimTemplates:
    - metadata:
        name: kafka-data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi

Step 6 — Initial Deployment

Apply all manifests:

kubectl apply -f namespace.yaml
kubectl apply -f kafka-svc.yaml
kubectl apply -f kafka-jaas.yaml
kubectl apply -f kafka-sasl.yaml
kubectl apply -f kafka-stateful-set.yaml

Watch the pods come up:

kubectl get pods -n kafka -w

All three pods should reach Running status within a minute or two. Check kafka-0 logs to confirm a clean startup:

kubectl logs -n kafka kafka-0

A healthy startup ends with:

[KafkaRaftServer nodeId=0] Kafka Server started

Step 7 — Bootstrap SCRAM Credentials

This is the most involved post-deployment step, and also the most commonly misunderstood. Here is why it requires a special process.

The Chicken-and-Egg Problem

The broker's INTERNAL listener (port 9092) requires SASL/SCRAM-SHA-512 authentication. The kafka-configs.sh admin tool needs to connect to a broker to write SCRAM credentials into Kafka's metadata. But to connect to the broker, you need valid SCRAM credentials — which don't exist yet because you haven't written them.

There are two apparent escape hatches that do not actually work:

Using the JAAS file credentials directly against port 9092 — the JAAS file tells the broker process what credentials to use for its own inter-broker connections, but those credentials must also exist in Kafka's metadata store before the SCRAM handshake will succeed. The JAAS file does not pre-populate metadata.
Using --bootstrap-controller against port 9093 — Kafka 4.x does not support the alterUserScramCredentials admin API via the controller quorum endpoint. It must go through a broker.

The solution is to temporarily open an unauthenticated PLAINTEXT listener on a separate port (9094), use it to write the credentials, then close it again.

7a — Add the Temporary Listener to the Service

Edit kafka-svc.yaml to add port 9094:

# kafka-svc.yaml (temporary — port 9094 will be removed after bootstrap)
apiVersion: v1
kind: Service
metadata:
  name: kafka-headless
  namespace: kafka
spec:
  clusterIP: None
  selector:
    app: kafka
  ports:
    - name: internal
      port: 9092
    - name: controller
      port: 9093
    - name: plaintext-bootstrap
      port: 9094

7b — Add the Temporary Listener to the StatefulSet

Edit kafka-stateful-set.yaml and make these changes to the broker container:

Add containerPort: 9094 to the ports list.

Change KAFKA_LISTENERS to:

- name: KAFKA_LISTENERS
  value: "INTERNAL://:9092,CONTROLLER://:9093,PLAINTEXT://:9094"

Change KAFKA_LISTENER_SECURITY_PROTOCOL_MAP to:

- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
  value: "INTERNAL:SASL_PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT"

Change KAFKA_ADVERTISED_LISTENERS to:

- name: KAFKA_ADVERTISED_LISTENERS
  value: "INTERNAL://$(POD_NAME).kafka-headless.kafka.svc.cluster.local:9092,PLAINTEXT://$(POD_NAME).kafka-headless.kafka.svc.cluster.local:9094"

7c — Apply and Wait for Rollout

kubectl apply -f kafka-svc.yaml
kubectl apply -f kafka-stateful-set.yaml
kubectl rollout status statefulset/kafka -n kafka

7d — Register the SCRAM Credentials

kubectl exec -n kafka kafka-0 -- \
  /opt/kafka/bin/kafka-configs.sh \
  --bootstrap-server kafka-0.kafka-headless.kafka.svc.cluster.local:9094 \
  --alter \
  --add-config 'SCRAM-SHA-512=[password=supersecret]' \
  --entity-type users \
  --entity-name admin

Expected output:

Completed updating config for user admin.

If you need additional users (application service accounts), register them now while port 9094 is still open:

kubectl exec -n kafka kafka-0 -- \
  /opt/kafka/bin/kafka-configs.sh \
  --bootstrap-server kafka-0.kafka-headless.kafka.svc.cluster.local:9094 \
  --alter \
  --add-config 'SCRAM-SHA-512=[password=apppassword]' \
  --entity-type users \
  --entity-name myapp

7e — Remove the Temporary Listener

Revert kafka-svc.yaml to remove port 9094:

# kafka-svc.yaml (final)
apiVersion: v1
kind: Service
metadata:
  name: kafka-headless
  namespace: kafka
spec:
  clusterIP: None
  selector:
    app: kafka
  ports:
    - name: internal
      port: 9092
    - name: controller
      port: 9093

Revert the StatefulSet changes — remove the PLAINTEXT entries from KAFKA_LISTENERS, KAFKA_LISTENER_SECURITY_PROTOCOL_MAP, and KAFKA_ADVERTISED_LISTENERS, and remove the containerPort: 9094 line.

Apply and wait:

kubectl apply -f kafka-svc.yaml
kubectl apply -f kafka-stateful-set.yaml
kubectl rollout status statefulset/kafka -n kafka

Step 8 — Verify the Cluster

Check the KRaft quorum status:

kubectl exec -n kafka kafka-0 -- \
  /opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-controller kafka-0.kafka-headless.kafka.svc.cluster.local:9093 \
  describe --status

A healthy cluster shows:

LeaderId:             0
LeaderEpoch:          1
HighWatermark:        303
MaxFollowerLag:       0
MaxFollowerLagTimeMs: 0
CurrentVoters:        [{"id": 0, ...}, {"id": 1, ...}, {"id": 2, ...}]
CurrentObservers:     []

MaxFollowerLag: 0 confirms all three nodes are fully in sync. Three voters and no observers confirms all nodes are full participants in the quorum.

Verify SASL authentication works on port 9092:

kubectl exec -n kafka kafka-0 -- bash -c '
cat > /tmp/client.properties << EOF
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="supersecret";
EOF
/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka-0.kafka-headless.kafka.svc.cluster.local:9092 \
  --command-config /tmp/client.properties \
  --list
'

Create a test topic and produce/consume a message:

kubectl exec -n kafka kafka-0 -- bash -c '
cat > /tmp/client.properties << EOF
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="supersecret";
EOF

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka-0.kafka-headless.kafka.svc.cluster.local:9092 \
  --command-config /tmp/client.properties \
  --create --topic test --partitions 3 --replication-factor 3

echo "hello kafka" | /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server kafka-0.kafka-headless.kafka.svc.cluster.local:9092 \
  --producer.config /tmp/client.properties \
  --topic test

/opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server kafka-0.kafka-headless.kafka.svc.cluster.local:9092 \
  --consumer.config /tmp/client.properties \
  --topic test --from-beginning --max-messages 1
'

Production Hardening Checklist

The cluster at this point is functional and secured with SASL. The following additional steps are needed before handling real production traffic.

Add TLS Encryption

The current setup uses SASL_PLAINTEXT, meaning credentials and data are transmitted unencrypted within the cluster. For any environment where network traffic could be intercepted, upgrade to SASL_SSL.

Generate a keystore and truststore using keytool. The Subject Alternative Names (SANs) must cover all broker hostnames — this is the most common mistake when setting up TLS for Kafka on Kubernetes.

PASS=yourpassword
SAN="dns:kafka-0.kafka-headless.kafka.svc.cluster.local,\
dns:kafka-1.kafka-headless.kafka.svc.cluster.local,\
dns:kafka-2.kafka-headless.kafka.svc.cluster.local,\
dns:kafka-headless.kafka.svc.cluster.local"

# Generate CA
keytool -genkeypair -alias ca -keyalg RSA -keysize 2048 \
  -dname "CN=kafka-ca" -validity 3650 \
  -keystore ca.jks -storepass $PASS -storetype JKS

keytool -exportcert -alias ca -keystore ca.jks \
  -storepass $PASS -file ca.crt

# Generate broker keypair and CSR
keytool -genkeypair -alias kafka -keyalg RSA -keysize 2048 \
  -dname "CN=kafka" -validity 3650 \
  -keystore keystore.jks -storepass $PASS -storetype JKS

keytool -certreq -alias kafka -keystore keystore.jks \
  -storepass $PASS -file kafka.csr

# Sign the CSR with the CA, including all broker SANs
keytool -gencert -alias ca -keystore ca.jks -storepass $PASS \
  -infile kafka.csr -outfile kafka.crt -validity 3650 \
  -ext "SAN=$SAN"

# Import CA + signed cert into the keystore
keytool -importcert -alias ca -file ca.crt \
  -keystore keystore.jks -storepass $PASS -noprompt
keytool -importcert -alias kafka -file kafka.crt \
  -keystore keystore.jks -storepass $PASS -noprompt

# Build truststore containing only the CA
keytool -importcert -alias ca -file ca.crt \
  -keystore truststore.jks -storepass $PASS -noprompt

# Store in Kubernetes
kubectl create secret generic kafka-tls -n kafka \
  --from-file=keystore.jks=keystore.jks \
  --from-file=truststore.jks=truststore.jks \
  --from-literal=password=$PASS \
  --dry-run=client -o yaml | kubectl apply -f -

Then update the StatefulSet to mount the secret and change the listener protocols:

INTERNAL:SASL_PLAINTEXT → INTERNAL:SASL_SSL
CONTROLLER:PLAINTEXT → CONTROLLER:SSL
Add KAFKA_SSL_KEYSTORE_LOCATION, KAFKA_SSL_TRUSTSTORE_LOCATION, KAFKA_SSL_KEYSTORE_PASSWORD, KAFKA_SSL_TRUSTSTORE_PASSWORD env vars
Mount the kafka-tls secret as a volume at /tls

Change Default Credentials

The admin/supersecret credentials used in this guide are for demonstration only. Before going to production:

Update kafka-jaas.yaml with a strong randomly generated password
Update kafka-sasl.yaml with the same password
Re-register the SCRAM credentials using the bootstrap process in Step 7
Rotate any client configurations that reference the old credentials

Network Policies

Add a NetworkPolicy to restrict which pods can reach Kafka:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: kafka-network-policy
  namespace: kafka
spec:
  podSelector:
    matchLabels:
      app: kafka
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: kafka
      ports:
        - port: 9092
        - port: 9093
    - from:
        - namespaceSelector: {}
      ports:
        - port: 9092
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: kafka

Pod Anti-Affinity

Prevent Kubernetes from scheduling multiple Kafka pods on the same node:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
            - key: app
              operator: In
              values:
                - kafka
        topologyKey: kubernetes.io/hostname

Add this under spec.template.spec in the StatefulSet.

Pod Disruption Budget

Ensure Kubernetes never takes down more than one Kafka pod at a time during voluntary disruptions such as node drains or rolling updates:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: kafka-pdb
  namespace: kafka
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: kafka

Liveness and Readiness Probes

Add probes so Kubernetes can detect and restart unhealthy pods:

livenessProbe:
  tcpSocket:
    port: 9092
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 6

readinessProbe:
  tcpSocket:
    port: 9092
  initialDelaySeconds: 20
  periodSeconds: 5
  failureThreshold: 3

Resource Tuning

Workload	CPU Request	Memory Request	Storage
Development	250m	512Mi	5Gi
Low traffic	500m	1Gi	20Gi
Medium traffic	1	2Gi	50Gi
High traffic	2–4	4–8Gi	100Gi+

JVM Heap Tuning

Set the JVM heap via the KAFKA_HEAP_OPTS env var. A common rule of thumb is 50% of the container memory limit, not exceeding 6Gi:

- name: KAFKA_HEAP_OPTS
  value: "-Xms2g -Xmx2g"

Monitoring

The official apache/kafka image exposes JMX metrics. Enable them with:

- name: KAFKA_JMX_PORT
  value: "9999"
- name: KAFKA_JMX_HOSTNAME
  value: "0.0.0.0"

Key metrics to monitor:

kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions — should always be 0
kafka.controller:type=KafkaController,name=ActiveControllerCount — should be 1 across the cluster
kafka.network:type=RequestMetrics,name=TotalTimeMs — producer/consumer latency
JVM GC pause times — long pauses indicate heap needs tuning

Common Troubleshooting

Pod stuck in Init state

Check the init container logs:

kubectl logs -n kafka kafka-0 -c format-storage
kubectl logs -n kafka kafka-0 -c init-node-id

No readable meta.properties error

The broker cannot find formatted storage. Ensure you are using /etc/kafka/docker/run as the entrypoint, not kafka-server-start.sh. The latter ignores KAFKA_* env vars and the broker looks in the wrong directory.

Invalid cluster.id error

The CLUSTER_ID env var does not match what was written to meta.properties during the format step. Delete the PVCs to force a reformat:

kubectl delete pvc -n kafka -l app=kafka

There must be at least one broker advertised listener during format

The temp kraft.properties used in the format-storage init container must include a non-controller listener. Ensure listeners, advertised.listeners, listener.security.protocol.map, and inter.broker.listener.name are all present in the temp properties file.

SASL authentication timeout on port 9092

The SCRAM credentials have not been registered. The JAAS file alone is not enough — run the bootstrap process in Step 7 to write the credentials into Kafka's metadata store.

UnsupportedEndpointTypeException when using --bootstrap-controller

The alterUserScramCredentials admin API is not supported via the controller quorum endpoint in Kafka 4.x. You must connect through a broker (port 9092). Use the temporary PLAINTEXT listener on port 9094 as described in Step 7.

Port 9094 connection refused

The headless service does not expose port 9094 by default. You must explicitly add it to kafka-svc.yaml during the bootstrap process (Step 7a) and remove it afterwards.

Brokers cannot reach each other

Verify the headless service exists and pod DNS resolves from within a pod:

kubectl exec -n kafka kafka-0 -- \
  nslookup kafka-1.kafka-headless.kafka.svc.cluster.local

MaxFollowerLag is non-zero

One broker is behind. Check its logs for errors. Common causes are disk I/O pressure, a recent pod restart, or network instability.

Upgrading Kafka

To upgrade to a new Kafka version, update the image field in the StatefulSet. Kubernetes performs a rolling restart one pod at a time. Since KRaft requires a majority, the cluster stays available as long as at least 2 of 3 pods are running.

kubectl set image statefulset/kafka \
  kafka=apache/kafka:4.3.0 \
  -n kafka

kubectl rollout status statefulset/kafka -n kafka

Summary

Step	What it does
namespace.yaml	Isolates all Kafka resources
kafka-svc.yaml	Headless service for stable pod DNS
kafka-jaas.yaml	JAAS config for inter-broker SCRAM auth
kafka-sasl.yaml	Secret for client applications
kafka-stateful-set.yaml	Brokers, init containers, storage, config
Step 7 — SCRAM bootstrap	Temporarily open port 9094, write credentials to metadata, close port 9094

The cluster is production-ready when TLS is enabled, credentials are rotated to strong values, pod anti-affinity and a PodDisruptionBudget are in place, resource limits are tuned for your workload, and monitoring is active.

How GNOME drives me Crazy on Ubuntu

giveitatry — Fri, 20 Feb 2026 19:22:06 +0000

If you’re using Ubuntu with the default GNOME desktop, you’ve probably experienced this:

You click a pinned app in the dock (for example Firefox), and instead of switching to your already open window… GNOME shows you thumbnails of all open windows.

This makes me crazy. I need additional click to switch the app.

I don’t want previews.
I just want it to switch to the next window.

Luckily, Ubuntu makes this easy to fix.

Why This Happens

Ubuntu uses a modified GNOME dock called Ubuntu Dock, which is based on Dash to Dock.

By default, clicking a pinned app:

Shows window previews if multiple windows are open.

But we can change that behavior.

The Fix: Make Click Cycle Through Windows

Step 1 – Open Dock Settings

Run this command in Terminal:

gnome-extensions prefs ubuntu-dock@ubuntu.com

Step 2 – Change Click Behavior

Go to the Behavior tab.
Find Click action
Change it to:

Cycle through windows

That’s it.

How to Add a Subdomain with Hetzner DNS API

giveitatry — Sun, 01 Feb 2026 18:19:42 +0000

Hetzner’s DNS Public API allows you to programmatically manage your domains, including creating subdomains. A subdomain is simply a DNS record under your main zone.

1. Get Your Zone ID

Each domain in Hetzner DNS has a unique zone_id. To find it:

curl -X GET "https://dns.hetzner.com/api/v1/zones" \
  -H "Auth-API-Token: $API_TOKEN"

Look for your domain in the response and note its id. For example:

{
  "id": "ABC...",
  "name": "domain.com"
}

2. Add the Subdomain Record

Once you have the zone_id, you can create a subdomain by adding a DNS record. For example, to create xxx.domain.com pointing to 1.2.3.4:

curl -X POST "https://dns.hetzner.com/api/v1/records" \
  -H "Auth-API-Token: $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "zone_id": "ABC...",
    "type": "A",
    "name": "xxx",
    "value": "1.2.3.4",
    "ttl": 300
  }'

Parameters:

zone_id → The zone where the subdomain will be added.
type → DNS record type (A, CNAME, MX, etc.).
name → Subdomain prefix (xxx).
value → IP address or target domain (1.2.3.4 here).
ttl → Time-to-live in seconds (optional, default 300).

3. Verify the Subdomain

To confirm the subdomain was created:

curl -X GET "https://dns.hetzner.com/api/v1/records?zone_id=ABC..." \
  -H "Auth-API-Token: $API_TOKEN"

Check for your new record in the response.

✅ Summary

Subdomains are just DNS records in Hetzner.
Use GET /zones to find your zone_id.
Use POST /records to add subdomains.
Use GET /records to list or verify existing subdomains.

This approach lets you automate DNS management for your domains using Hetzner’s DNS API.

How to Clone a GitLab Repository with a Self-Signed Certificate

giveitatry — Sun, 01 Feb 2026 12:55:26 +0000

When working with a GitLab instance using a self-signed SSL certificate, attempting to clone a repository over HTTPS often fails with an error like:

fatal: unable to access 'https://gitlab.example.com/group/project.git/': 
server certificate verification failed. CAfile: none CRLfile: none

This happens because Git does not trust self-signed certificates by default. Below are safe, step-by-step ways to fix it.

1. Understanding the Problem

Git verifies SSL certificates to ensure the server you’re connecting to is legitimate. Self-signed certificates aren’t trusted by default because they’re not signed by a recognized Certificate Authority (CA).

This means HTTPS connections fail unless Git is explicitly told to trust the certificate.

2. Export the Self-Signed Certificate

Option A: Export via Chrome

Open your GitLab URL in Chrome:

https://gitlab.example.com

Click the lock icon in the address bar → Connection is secure → Certificate is valid.
Go to the Details tab → Click Export….
Save the certificate as gitlab-selfsigned.crt in a permanent location:

Windows: C:\certs\gitlab-selfsigned.crt
Linux/macOS: /home/username/certs/gitlab-selfsigned.crt

Option B: Export via Firefox

Firefox doesn’t always allow direct export from the page, but here are two reliable ways:

Method 1: Using Firefox Preferences

Open Firefox and go to:

about:preferences#privacy

Scroll down to Certificates → Click View Certificates.
Go to the Servers tab, find your GitLab domain (gitlab.example.com), select it → Export…
Save as gitlab-selfsigned.crt.

Method 2: Using OpenSSL (cross-platform, recommended)

Open a terminal (Linux, macOS, or Git Bash on Windows) and run:

echo | openssl s_client -connect gitlab.example.com:443 -showcerts 2>/dev/null | openssl x509 -outform PEM > gitlab-selfsigned.crt

This fetches the certificate directly from GitLab.
Saves it as gitlab-selfsigned.crt in the current directory.

3. Configure Git to Trust the Certificate

Once you have the .crt file:

Windows:

git config --global http.sslCAInfo "C:/certs/gitlab-selfsigned.crt"

Linux / macOS:

git config --global http.sslCAInfo "/home/username/certs/gitlab-selfsigned.crt"

Verify:

git config --global --get http.sslCAInfo

4. Clone the Repository

Now, you can securely clone your repository over HTTPS:

git clone https://gitlab.example.com/group/project.git

5. Alternative: Use SSH Instead of HTTPS

SSH avoids all certificate issues:

Generate an SSH key if you don’t have one:

ssh-keygen -t ed25519 -C "your_email@example.com"

Copy the public key (~/.ssh/id_ed25519.pub) and add it to GitLab:
User Settings → SSH Keys → Add Key
Clone via SSH:

git clone git@gitlab.example.com:group/project.git

No SSL certificate issues occur using SSH.

6. Quick & Unsafe Workaround (Testing Only)

You can temporarily disable SSL verification:

git config --global http.sslVerify false

⚠️ Warning: This is insecure. It allows MITM attacks and should never be used permanently.

7. PyCharm Considerations

PyCharm may use a separate Git executable. To ensure it works:

Go to:

File → Settings → Version Control → Git

Check the Path to Git executable.
Click Test.
Make sure this Git has the http.sslCAInfo configuration set (or use SSH instead).

8. Summary

Recommended workflow for self-signed GitLab:

Export the self-signed certificate (Chrome or Firefox/OpenSSL).
Configure Git to trust the certificate.
Clone via HTTPS.

Alternative: Use SSH to avoid certificate issues entirely.

Avoid: disabling SSL verification permanently — it is insecure.

How to Clone a GitLab Repository with a Self-Signed Certificate

giveitatry — Sun, 01 Feb 2026 08:52:44 +0000

When working with a GitLab instance using a self-signed SSL certificate, attempting to clone a repository over HTTPS often fails with an error like:

fatal: unable to access 'https://gitlab.example.com/group/project.git/': 
server certificate verification failed. CAfile: none CRLfile: none

This happens because Git does not trust self-signed certificates by default. Below are safe, step-by-step ways to fix it.

1. Understanding the Problem

This means HTTPS connections fail unless Git is explicitly told to trust the certificate.

2. Export the Self-Signed Certificate

Option A: Export via Chrome

Open your GitLab URL in Chrome:

https://gitlab.example.com

Click the lock icon in the address bar → Connection is secure → Certificate is valid.
Go to the Details tab → Click Export….
Save the certificate as gitlab-selfsigned.crt in a permanent location:

Windows: C:\certs\gitlab-selfsigned.crt
Linux/macOS: /home/username/certs/gitlab-selfsigned.crt

Option B: Export via Firefox

Firefox doesn’t always allow direct export from the page, but here are two reliable ways:

Method 1: Using Firefox Preferences

Open Firefox and go to:

about:preferences#privacy

Scroll down to Certificates → Click View Certificates.
Go to the Servers tab, find your GitLab domain (gitlab.example.com), select it → Export…
Save as gitlab-selfsigned.crt.

Method 2: Using OpenSSL (cross-platform, recommended)

Open a terminal (Linux, macOS, or Git Bash on Windows) and run:

echo | openssl s_client -connect gitlab.example.com:443 -showcerts 2>/dev/null | openssl x509 -outform PEM > gitlab-selfsigned.crt

This fetches the certificate directly from GitLab.
Saves it as gitlab-selfsigned.crt in the current directory.

3. Configure Git to Trust the Certificate

Once you have the .crt file:

Windows:

git config --global http.sslCAInfo "C:/certs/gitlab-selfsigned.crt"

Linux / macOS:

git config --global http.sslCAInfo "/home/username/certs/gitlab-selfsigned.crt"

Verify:

git config --global --get http.sslCAInfo

4. Clone the Repository

Now, you can securely clone your repository over HTTPS:

git clone https://gitlab.example.com/group/project.git

5. Alternative: Use SSH Instead of HTTPS

SSH avoids all certificate issues:

Generate an SSH key if you don’t have one:

ssh-keygen -t ed25519 -C "your_email@example.com"

Copy the public key (~/.ssh/id_ed25519.pub) and add it to GitLab:
User Settings → SSH Keys → Add Key
Clone via SSH:

git clone git@gitlab.example.com:group/project.git

No SSL certificate issues occur using SSH.

6. Quick & Unsafe Workaround (Testing Only)

You can temporarily disable SSL verification:

git config --global http.sslVerify false

⚠️ Warning: This is insecure. It allows MITM attacks and should never be used permanently.

7. PyCharm Considerations

PyCharm may use a separate Git executable. To ensure it works:

Go to:

File → Settings → Version Control → Git

Check the Path to Git executable.
Click Test.
Make sure this Git has the http.sslCAInfo configuration set (or use SSH instead).

8. Summary

Recommended workflow for self-signed GitLab:

Export the self-signed certificate (Chrome or Firefox/OpenSSL).
Configure Git to trust the certificate.
Clone via HTTPS.

Alternative: Use SSH to avoid certificate issues entirely.

Avoid: disabling SSL verification permanently — it is insecure.

Running TeamCity Agents on Bare Metal Linux

giveitatry — Sat, 24 Jan 2026 21:54:16 +0000

This article describes how to install and configure a TeamCity Build Agent on bare-metal Linux, with Java and Docker support, so the agent can run Docker-based builds reliably.

The guide assumes:

Linux (Ubuntu/Debian-based)
Root or sudo access
TeamCity Server already running

1. System Requirements

Hardware

CPU: 2 cores minimum (4+ recommended)
RAM: 4 GB minimum (8+ recommended for Docker builds)
Disk: ≥ 20 GB free

Software

Linux (Ubuntu 20.04+ recommended)
Java Runtime Environment
Docker Engine
TeamCity Build Agent

2. Update the System

Before installing anything, update the package index:

sudo apt update
sudo apt upgrade -y

3. Install Java (Required by TeamCity Agent)

TeamCity agents require Java to run.

Install the default JRE:

sudo apt install -y default-jre

Verify installation:

java -version

Expected output:

openjdk version "11.x" or newer

4. Install Docker Engine

Docker is required if the agent will run Docker builds.

Install Docker from Ubuntu repositories:

sudo apt install -y docker.io

Enable and start Docker:

sudo systemctl enable docker
sudo systemctl start docker

Verify Docker is running:

sudo docker ps

5. Allow the Agent User to Run Docker

Docker access is controlled via the Unix socket:

/var/run/docker.sock

Only users in the docker group can access it.

Add the agent user to the docker group

Example (agent user = risto):

sudo usermod -aG docker risto

⚠️ Important:
Group membership changes apply only after logout/login.

➡️ Log out and log back in (or reboot the machine).

Verify:

groups

You must see:

docker

Test Docker access without sudo:

docker ps

6. Install the TeamCity Build Agent

6.1 Download the Agent Package

On the agent machine:

mkdir -p /opt/teamcity-agent
cd /opt/teamcity-agent

Download from your TeamCity server:

wget http://<TEAMCITY_SERVER>:8111/update/buildAgent.zip

Unpack:

unzip buildAgent.zip

7. Configure the Build Agent

7.1 Configure Server Connection

Edit the agent configuration file:

nano conf/buildAgent.properties

Set the TeamCity server URL:

serverUrl=http://<TEAMCITY_SERVER>:8111

(Optional but recommended)

name=baremetal-agent-01
workDir=../work
tempDir=../temp
systemDir=../system

7.2 Docker Detection (Important)

TeamCity automatically detects Docker at agent startup.

Requirements:

Docker installed
Docker daemon running
Agent user can run docker ps

No extra Docker configuration is required if permissions are correct.

8. Start the Agent

Run the agent manually the first time:

bin/agent.sh start

Check logs:

tail -f logs/teamcity-agent.log

9. Authorize the Agent in TeamCity

In the TeamCity web UI:

Agents → Unauthorized

Authorize the new agent.

Once authorized, verify agent parameters:

Agents → <agent> → Parameters → System Properties

You should see:

docker.server.osType = linux
docker.server.version = 26.x
docker.client.version = 26.x
java.runtime.version = 11.x

10. Common Problems and Fixes

❌ Docker not detected by TeamCity

Cause: Agent started before Docker was installed or permissions fixed.

Fix:

sudo systemctl restart docker
bin/agent.sh restart

❌ `permission denied /var/run/docker.sock`

Cause: Agent user not in docker group or session not refreshed.

Fix:

sudo usermod -aG docker <agent-user>
# logout + login
bin/agent.sh restart

❌ Agent runs but Docker builds never start

Check unmet requirements:

docker.server.version exists
docker.server.osType contains linux

If missing → Docker not visible to agent.

11. Security Notes

Adding a user to the docker group grants root-equivalent access
Use dedicated agent users
Restrict agent responsibilities when possible

12. Summary

To run TeamCity agents on bare metal with Docker:

Install Java (default-jre)
Install Docker (docker.io)
Add agent user to docker group
Log out and back in
Configure buildAgent.properties
Start and authorize the agent

Complete Guide: Let's Encrypt SSL with cert-manager and Hetzner DNS on Kubernetes

giveitatry — Fri, 23 Jan 2026 21:32:35 +0000

Introduction

This guide walks you through setting up automatic SSL certificates from Let's Encrypt on a Kubernetes cluster using cert-manager with Hetzner DNS for DNS-01 challenge validation. This is the most reliable method for issuing and renewing SSL certificates automatically, especially if you have multiple domains or subdomains.

By the end of this guide, you'll have:

Automatic SSL certificate issuance via Let's Encrypt
DNS-based validation (no need to expose HTTP validation endpoints)
Automatic certificate renewal before expiration
HTTPS redirect middleware in Traefik

Prerequisites

A Kubernetes cluster (k3s, kubeadm, EKS, etc.)
kubectl configured to access your cluster
A domain with DNS hosted at Hetzner DNS
Helm installed on your local machine
Traefik ingress controller (this guide uses Traefik)

Architecture Overview

Here's how the process works:

You create a Certificate resource in Kubernetes
cert-manager reads this and contacts Let's Encrypt
Let's Encrypt returns a DNS challenge token
cert-manager calls the Hetzner webhook with this token
The Hetzner webhook uses the Hetzner DNS API to create a TXT record
Let's Encrypt queries the DNS and validates the TXT record
Let's Encrypt issues the certificate
cert-manager stores it in a Kubernetes Secret
Your Ingress uses this Secret for HTTPS

Step 1: Install cert-manager

cert-manager is the core component that manages certificate lifecycle.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.19.2/cert-manager.yaml

Wait for it to be ready:

kubectl wait --for=condition=ready pod -l app.kubernetes.io/instance=cert-manager -n cert-manager --timeout=300s

Verify installation:

kubectl get pods -n cert-manager

You should see three pods running:

cert-manager
cert-manager-cainjector
cert-manager-webhook

Step 2: Create Hetzner DNS API Token

Log in to Hetzner DNS Control Panel
Navigate to "API Tokens" (usually under account settings)
Click "Create new token"
Give it a name like "cert-manager"
Copy the token immediately (you won't see it again)

Important: Keep this token secret. Anyone with this token can modify your DNS records.

Step 3: Install the Hetzner Webhook for cert-manager

The Hetzner webhook is a cert-manager extension that knows how to talk to the Hetzner DNS API.

Add the Helm repository:

helm repo add cert-manager-webhook-hetzner https://vadimkim.github.io/cert-manager-webhook-hetzner
helm repo update

Install the webhook:

helm install cert-manager-webhook-hetzner \
  --namespace cert-manager \
  cert-manager-webhook-hetzner/cert-manager-webhook-hetzner

Verify the webhook pod is running:

kubectl get pods -n cert-manager | grep hetzner

Step 4: Create the Hetzner API Secret

Create a file called hetzner-secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: hcloud-api-token-secret
  namespace: cert-manager
type: Opaque
stringData:
  api-key: "your-hetzner-api-token-here"

Important: The key must be named api-key (not token). This is what the webhook expects.

Apply it:

kubectl apply -f hetzner-secret.yaml

Verify the secret was created:

kubectl get secret hcloud-api-token-secret -n cert-manager

Step 5: Verify the Webhook's API Group Registration

Before creating the ClusterIssuer, you need to know the exact API group name that the webhook registered. This is crucial and often a source of problems.

Run this command:

kubectl api-resources | grep hetzner

You should see output like:

hetzner    hetzner.cert-mananger-webhook.noshoes.xyz/v1alpha1    false    ChallengePayload

Note the exact API group name - in this example it's hetzner.cert-mananger-webhook.noshoes.xyz. This is what you'll use in your ClusterIssuer. Pay attention to the spelling - some versions have a typo: "mananger" instead of "manager".

Step 6: Set Up RBAC Permissions

cert-manager needs permission to call the webhook, and the webhook needs permission to read the Hetzner API secret.

Create a file called rbac.yaml:

# Allow webhook to read the Hetzner API token
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cert-manager-webhook-hetzner-secret-access
  namespace: cert-manager
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["hcloud-api-token-secret"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cert-manager-webhook-hetzner-secret-access
  namespace: cert-manager
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cert-manager-webhook-hetzner-secret-access
subjects:
- kind: ServiceAccount
  name: cert-manager-webhook-hetzner
  namespace: cert-manager
---
# Allow cert-manager to call the webhook
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cert-manager-webhook-hetzner
rules:
- apiGroups:
  - hetzner.cert-mananger-webhook.noshoes.xyz
  resources:
  - hetzner
  verbs:
  - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cert-manager-webhook-hetzner
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cert-manager-webhook-hetzner
subjects:
- kind: ServiceAccount
  name: cert-manager
  namespace: cert-manager

Important: Replace hetzner.cert-mananger-webhook.noshoes.xyz in the ClusterRole with whatever kubectl api-resources | grep hetzner showed you.

Apply it:

kubectl apply -f rbac.yaml

Step 7: Create the ClusterIssuer

The ClusterIssuer tells cert-manager how to request certificates from Let's Encrypt using Hetzner DNS.

Create a file called cluster-issuer.yaml:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dns-hetzner
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-dns-hetzner-key
    solvers:
      - dns01:
          webhook:
            groupName: hetzner.cert-mananger-webhook.noshoes.xyz
            solverName: hetzner
            config:
              secretName: hcloud-api-token-secret
              zoneName: example.com
              apiUrl: https://dns.hetzner.com/api/v1

Replace:

admin@example.com with your email
groupName with the exact value from kubectl api-resources | grep hetzner
zoneName with your root domain (e.g., example.com)

For testing/staging (to avoid Let's Encrypt rate limits), use this server instead:

server: https://acme-staging-v02.api.letsencrypt.org/directory

Apply it:

kubectl apply -f cluster-issuer.yaml

Verify the issuer is ready:

kubectl describe clusterissuer letsencrypt-dns-hetzner

Look for Status: True in the conditions.

Step 8: Create Certificate Resources

Now create a Certificate resource for each domain you want to secure.

Create a file called certificates.yaml:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example-tls
  namespace: default
spec:
  secretName: example-tls
  issuerRef:
    name: letsencrypt-dns-hetzner
    kind: ClusterIssuer
  commonName: example.com
  dnsNames:
    - example.com
    - "*.example.com"
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-tls
  namespace: default
spec:
  secretName: api-tls
  issuerRef:
    name: letsencrypt-dns-hetzner
    kind: ClusterIssuer
  commonName: api.example.com
  dnsNames:
    - api.example.com

Replace:

Namespace with your application's namespace
Domain names with your actual domains
Certificate names as appropriate

Apply it:

kubectl apply -f certificates.yaml

Step 9: Monitor Certificate Creation

Watch the certificates being created:

kubectl get certificate -A -w

Check detailed status:

kubectl describe certificate example-tls -n default

Watch the challenges:

kubectl get challenge -n default -w

Check the webhook logs to see DNS records being created:

kubectl logs -n cert-manager -l app=cert-manager-webhook-hetzner -f --tail=50

You should see messages like:

Added TXT record result: {"record":{"id":"...","type":"TXT","name":"_acme-challenge.example",...}}
Presented txt record _acme-challenge.example.com.
Delete TXT record result: {"error":{}}

Once the certificate shows Ready: True, the secret is ready to use.

Step 10: Use Certificates in Traefik Ingress

For standard Ingress with Traefik:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  namespace: default
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-dns-hetzner
spec:
  ingressClassName: traefik
  tls:
    - hosts:
        - example.com
      secretName: example-tls
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-service
                port:
                  number: 80

For Traefik IngressRoute with HTTP-to-HTTPS redirect:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: redirect-https
  namespace: default
spec:
  redirectScheme:
    scheme: https
    permanent: true
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: example-http
  namespace: default
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`example.com`)
      kind: Rule
      middlewares:
        - name: redirect-https
          namespace: default
      services:
        - name: my-service
          port: 80
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: example-https
  namespace: default
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`example.com`)
      kind: Rule
      services:
        - name: my-service
          port: 80
  tls:
    secretName: example-tls

Troubleshooting Guide

Problem 1: Certificate Stuck in "Pending" State

Symptom: kubectl get certificate shows READY: False for hours

Solution:

Check the certificate details:

kubectl describe certificate example-tls -n default

Look at the events section. The most common causes are listed below.

Problem 2: "Unable to get secret" Error

Error message:

unable to get secret `hcloud-api-token-secret/cert-manager`; secrets "hcloud-api-token-secret" is forbidden

Cause: The RBAC permissions are not set up correctly. The webhook's service account can't read the Hetzner API token secret.

Solution:

Verify the secret exists:

kubectl get secret hcloud-api-token-secret -n cert-manager

Reapply the RBAC rules:

kubectl apply -f rbac.yaml

Make sure the Role has the correct secret name:

rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["hcloud-api-token-secret"]  # Exact match required
  verbs: ["get"]

Problem 3: "Key 'api-key' not found in secret"

Error message:

unable to get api-key from secret `hcloud-api-token-secret`; key "api-key" not found in secret data

Cause: The secret key is named wrong. The webhook expects exactly api-key.

Solution:

Delete and recreate the secret with the correct key name:

kubectl delete secret hcloud-api-token-secret -n cert-manager

Create it again with api-key:

apiVersion: v1
kind: Secret
metadata:
  name: hcloud-api-token-secret
  namespace: cert-manager
type: Opaque
stringData:
  api-key: "your-token-here"

Problem 4: "cannot create resource 'hetzner' is forbidden"

Error message:

User "system:serviceaccount:cert-manager:cert-manager" cannot create resource "hetzner" in API group "hetzner.cert-manager-webhook.noshoes.xyz"

Cause: cert-manager service account doesn't have permission to call the webhook.

Solution:

Reapply the ClusterRole and ClusterRoleBinding:

kubectl apply -f rbac.yaml

Verify the ClusterRoleBinding exists:

kubectl get clusterrolebinding | grep hetzner

Problem 5: "the server could not find the requested resource"

Error message:

the server could not find the requested resource (post hetzner.cert-manager-webhook.noshoes.xyz)

Cause: The groupName in your ClusterIssuer doesn't match what the webhook registered.

Solution:

Check the actual API group:

kubectl api-resources | grep hetzner

This outputs something like:

hetzner    hetzner.cert-mananger-webhook.noshoes.xyz/v1alpha1    false    ChallengePayload

Copy the exact API group (e.g., hetzner.cert-mananger-webhook.noshoes.xyz) and update your ClusterIssuer:

solvers:
  - dns01:
      webhook:
        groupName: hetzner.cert-mananger-webhook.noshoes.xyz  # Use exact match
        solverName: hetzner

Problem 6: The Webhook Has a Typo in the API Group Name

Issue: The webhook registers as hetzner.cert-mananger-webhook.noshoes.xyz (with "mananger" - missing 'e')

Cause: This is a known issue in some versions of the Hetzner webhook Helm chart.

Solution:

You have two options:

Option A (Quickest): Just use the misspelled group name in your ClusterIssuer. It works fine despite the typo:

groupName: hetzner.cert-mananger-webhook.noshoes.xyz

Option B (Proper fix): Reinstall the webhook with a corrected version or use a different webhook implementation that doesn't have this typo.

Problem 7: Challenge Not Being Presented

Symptom: Challenge status shows Presented: false but no errors in logs

Cause: Webhook pod might have crashed or become unhealthy

Solution:

Check webhook health:

kubectl get pods -n cert-manager | grep hetzner

Check pod logs:

kubectl logs -n cert-manager -l app=cert-manager-webhook-hetzner

Restart the webhook:

kubectl rollout restart deployment/cert-manager-webhook-hetzner -n cert-manager

Delete the challenge to trigger a retry:

kubectl delete challenge -n default --all

Problem 8: DNS Record Not Being Created in Hetzner

Symptom: No TXT record appears in Hetzner DNS console

Cause: Usually an invalid API token or zone name mismatch

Solution:

Verify the API token is correct:

kubectl get secret hcloud-api-token-secret -n cert-manager -o jsonpath='{.data.api-key}' | base64 -d

Verify the zone name matches your domain:

zoneName: example.com  # Must match your actual domain

Check the webhook logs for errors:

kubectl logs -n cert-manager -l app=cert-manager-webhook-hetzner -f | grep -i error

Test the API token manually:

curl -H "Auth-API-Token: your-token" https://dns.hetzner.com/api/v1/zones

Problem 9: Certificate Ready but HTTPS Still Shows Old/Invalid Certificate

Cause: Ingress is not using the new secret or old ingress resource is still active

Solution:

Verify the secret exists:

kubectl get secret example-tls -n default

Check the Ingress is referencing it:

kubectl describe ingress example-ingress -n default

If using both Ingress and IngressRoute, delete the old Ingress:

kubectl delete ingress example-ingress -n default

Wait for the load balancer to update (usually 30 seconds to 2 minutes)

Verification Checklist

Once everything is set up, verify with this checklist:

# 1. cert-manager is running
kubectl get pods -n cert-manager

# 2. Webhook is running
kubectl get pods -n cert-manager | grep hetzner

# 3. ClusterIssuer is ready
kubectl describe clusterissuer letsencrypt-dns-hetzner

# 4. Certificate is ready
kubectl get certificate -A

# 5. Secret is created
kubectl get secret example-tls -n default

# 6. Ingress/IngressRoute is configured
kubectl get ingress,ingressroute -A

# 7. DNS records exist
nslookup _acme-challenge.example.com

# 8. HTTPS works
curl -I https://example.com

Automatic Renewal

cert-manager automatically renews certificates 30 days before expiration. You don't need to do anything.

Monitor renewal status:

kubectl describe certificate example-tls -n default

Check cert-manager logs for renewal activity:

kubectl logs -n cert-manager -l app=cert-manager | grep -i renew

Best Practices

Use ClusterIssuer, not Issuer - ClusterIssuer is namespace-agnostic and reusable
Use production Let's Encrypt - Only use staging for testing to avoid rate limits
Keep API token secret - Don't commit it to git, use sealed secrets or external secret management
Monitor certificate expiration - Set up alerts for certificates not renewing
Use separate secrets per domain - Makes it easier to rotate certs individually
Test in staging first - Use the Let's Encrypt staging server before production
Keep webhook updated - Regularly update the webhook Helm chart for security patches

Conclusion

You now have a fully automated SSL certificate system that:

Issues certificates automatically from Let's Encrypt
Validates using DNS (no need for HTTP validation)
Renews certificates automatically before expiration
Works across all your subdomains under one root domain
Integrates seamlessly with Traefik

If you encounter issues, the troubleshooting guide above covers the most common problems and their solutions.

Running TeamCity with a Docker-Capable Agent Using Docker Compose

giveitatry — Thu, 18 Dec 2025 09:00:51 +0000

TeamCity is a powerful CI/CD server from JetBrains, and running it inside Docker simplifies setup, upgrades, and isolation. This guide shows how to set up TeamCity Server and a TeamCity Agent capable of building Docker images.

Prerequisites

Before you start, ensure you have:

Docker installed and running
Docker Compose installed
Sufficient permissions to run Docker commands on your host
A terminal/command-line interface

Step 1: Create a directory structure

It’s best practice to separate your data, logs, and agent configuration:

mkdir -p teamcity/{data,logs,agent}
cd teamcity

Your directory structure should look like this:

teamcity/
├── docker-compose.yml
├── data/       # Server persistent data
├── logs/       # TeamCity logs
└── agent/      # Agent configuration

Step 2: Create the Docker Compose file

Create a file called docker-compose.yml with the following content:

version: "3.8"

services:
  teamcity-server:
    image: jetbrains/teamcity-server:2023.11.4
    container_name: teamcity-server
    ports:
      - "8111:8111"
    volumes:
      - ./data:/data/teamcity_server/datadir
      - ./logs:/opt/teamcity/logs
    restart: unless-stopped

  teamcity-agent:
    image: jetbrains/teamcity-agent:2023.11.4
    container_name: teamcity-agent
    user: root
    environment:
      - SERVER_URL=http://teamcity-server:8111
    volumes:
      - ./agent:/data/teamcity_agent/conf
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - teamcity-server
    restart: unless-stopped

Key points:

Server volumes:

./data stores TeamCity server data and build history.
./logs keeps server logs persistent across container restarts.

Agent setup:

The agent runs as root to have permissions to use the host Docker socket.
/var/run/docker.sock is mounted to allow the agent to access the host Docker daemon.
SERVER_URL points the agent to the server.

Step 3: Start TeamCity

Run:

docker compose up -d

-d runs containers in the background.
The server will be available at http://localhost:8111.
The agent container will automatically try to connect to the server.

Step 4: Initial TeamCity setup

Open your browser and go to:

http://localhost:8111

Follow the web setup wizard:

Confirm the data directory (mapped to ./data)
Choose a database (internal database is fine for testing)
Accept the license
Create the initial admin account

Step 5: Authorize the agent

Go to Agents → Unauthorized in the TeamCity UI.
Click Authorize for your agent.
Once authorized, the agent will show as Connected and Compatible.

Step 6: Verify Docker capability

Open a terminal inside the agent container:

docker exec -it teamcity-agent docker version

You should see the Docker client and server versions. If so, the agent can now build Docker images.

Step 7: Configure a Docker build in TeamCity

You can use Command Line or Docker runner build steps.

Example Command Line step:

docker build -t my-app:%build.number% .
docker push my-app:%build.number%

Example Docker Runner step:

Add a Docker build step in the UI.
Set the Dockerfile path.
Set the image name.
TeamCity will use the agent’s Docker access to build images.

Step 8: Recommended best practices

Keep your /data folder on fast, persistent storage.
Backup your TeamCity server data regularly.
Limit access to the TeamCity server and Docker socket, since running the agent as root gives it full control over Docker on the host.
For production environments, consider a Docker-in-Docker setup or a remote Docker host for better isolation.

Step 9: Stopping and restarting

To stop:

docker compose down

To restart:

docker compose up -d

The agent will reconnect automatically after restart.

✅ Summary

With this setup:

You have a fully functional TeamCity server running in Docker.
Your TeamCity agent can build Docker images using the host Docker daemon.
Data and logs persist across restarts, and you can expand with more agents if needed.

This setup is ideal for testing, development, and small production environments.

DEV Community: giveitatry

Upgrading K3s: Complete Guide

TL;DR

Your Current Situation

The Golden Rule: One Minor Version at a Time

Important: Traefik Version Change at v1.32

Important: etcd Warning for v1.34+

Method 1: Manual Upgrade via Install Script (Recommended)

Step 1 — Upgrade the server node, one minor version at a time

Step 2 — Upgrade each worker node

Step 3 — Verify after each hop

Method 2: Upgrade via Binary (Manual)

Draining Nodes Before Upgrading (For Production)

After the Full Upgrade

Summary

K3s Certificate Rotation: Complete Guide to Managing and Rotating Certificates

TL;DR — All Commands You Need

Certificate Types and Their Lifetimes

Client and Server Certificates — 365 days

CA (Certificate Authority) Certificates — 10 years

Automatic Certificate Renewal — How It Works

The Practical Implication

Warning Signs — What You See Before Expiry

What Happens When Certificates Actually Expire

Checking Certificate Status

Manual Rotation — Step by Step

On the Server Node

Rotating Specific Services Only

⚠️ Warning: Worker Nodes Will Disconnect After Rotation

What Happens

The Cascade With Local Storage

Fix — Restart the Agent on Every Worker Node

The Correct Order of Operations for a Full Cluster Rotation

CA Certificate Rotation

Non-disruptive vs. Disruptive Rotation

CA Rotation Procedure

Monitoring — Preventing Surprise Expiry

Summary Table

Key Takeaways

Running a TeamCity Agent as a systemd Service

📁 Prerequisites

⚙️ Create systemd service

🚀 Enable and start the service

🔍 Verify status

⚠️ Notes

Kafka 4.2.0 on Kubernetes - Complete Setup Guide - Exposed to Internet

Stage 1 — Bootstrap

1.1 namespace.yaml

1.2 kafka-jaas.yaml

1.3 kafka-sasl.yaml

1.4 traefik-config.yaml

1.5 kafka-traefik.yaml

1.6 kafka-stateful-bootstrap.yaml

1.7 Register SCRAM credentials

Stage 2 — Production (close port 9094)

2.1 kafka-stateful-final.yaml

2.2 Verify

2.3 Python client

Troubleshooting

Deploying Apache Kafka 4.2.0 on Kubernetes with KRaft, SASL, and High Availability

Overview

Prerequisites

Tools

Cluster Resources

Kubernetes Version

Architecture

File Structure

Step 1 — Namespace

Step 2 — Headless Service

Step 3 — JAAS Configuration

Step 4 — SASL Secret

Step 5 — StatefulSet

Why Two Init Containers?

Why /etc/kafka/docker/run Instead of kafka-server-start.sh?

Why CLUSTER_ID as an Env Var?

Generating Your Own Cluster ID

Step 6 — Initial Deployment

Step 7 — Bootstrap SCRAM Credentials

The Chicken-and-Egg Problem

7a — Add the Temporary Listener to the Service

Why `/etc/kafka/docker/run` Instead of `kafka-server-start.sh`?

Why `CLUSTER_ID` as an Env Var?