DEV Community: kamlesh merugu

Part 4: Automated Backups & Restore Procedures

kamlesh merugu — Wed, 14 Jan 2026 23:06:38 +0000

We will implement a robust, atomic backup strategy that protects your n8n workflows, credentials, Redis queues, and custom nodes.

Stack:

Target: PostgreSQL (n8n data), Redis (Queue state), N8N (Custom nodes & config)
Storage: Central Backup PVC (PG/Redis) + Direct Upload (N8N)
Cloud: Mega.nz (Off-site with 15-day retention)
Encryption: Optional GPG encryption

📋 Overview

Architecture: Central Jobs (DB) + Direct Job (N8N).
Atomic Staging: Jobs write to a staging folder first, then move to ready only when complete.
Destination: All backups go to k8s-backups on Mega (N8N goes into k8s-backups/n8n).
Encryption: Backups are encrypted before syncing to the cloud.

🧩 Step 1: Prepare Secrets

We need secrets in BOTH backup and prod namespaces.

1. Configure Rclone

On your local machine (or server), generate the config.

rclone config
# Name: mega
# Type: mega
# Account: Enter your Mega email/pass

2. Create Rclone Secrets

Create the secret in both namespaces.

# In Prod (for N8N Backup)
kubectl create secret generic rclone-secret \
  --from-file=rclone.conf=$HOME/.config/rclone/rclone.conf \
  -n prod

# In Backup (for Sync Job)
kubectl create secret generic rclone-secret \
  --from-file=rclone.conf=$HOME/.config/rclone/rclone.conf \
  -n backup

3. Create Encryption Secret (Backup NS)

Generates a key to encrypt database dumps.

openssl rand -base64 32 > backup-passphrase.txt
kubectl create secret generic backup-encryption-secret \
  --from-file=passphrase=backup-passphrase.txt \
  -n backup
# **Save this passphrase in a password manager!**
rm backup-passphrase.txt

4. Copy Database Password

The database secret is in prod, but the backup job runs in backup. We copy it while stripping internal IDs to prevent conflicts.

kubectl get secret postgres-secret -n prod -o yaml | \
sed -e 's/namespace: prod/namespace: backup/' \
    -e '/uid:/d' \
    -e '/resourceVersion:/d' \
    -e '/creationTimestamp:/d' | \
kubectl apply -f -

🧩 Step 2: Infrastructure

File: backups/01-infrastructure.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: backup
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: backup-pvc
  namespace: backup
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi

Apply:

kubectl apply -f backups/01-infrastructure.yaml

🧩 Step 3: Job 1 — PostgreSQL Backup

Dumps to staging, encrypts, moves to ready.

File: backups/02-postgres-backup.yaml

apiVersion: batch/v1
kind: CronJob
metadata:
  name: postgres-backup
  namespace: backup
spec:
  schedule: "0 2 * * *" 
  successfulJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup-client
            image: postgres:17-alpine
            command:
            - /bin/sh
            - -c
            - |
              set -e
              TIMESTAMP=$(date +%Y%m%d_%H%M%S)
              STAGING_DIR="/backup/staging/postgres"
              READY_DIR="/backup/ready/postgres"
              mkdir -p $STAGING_DIR $READY_DIR
              FILENAME="n8n_db_${TIMESTAMP}.dump"
              STAGING_FILE="${STAGING_DIR}/${FILENAME}"

              PGPASSWORD=$POSTGRES_PASSWORD pg_dump \
                -h postgres.prod.svc.cluster.local -U n8n -d n8n \
                -F c -b -f $STAGING_FILE

              if [ ! -s $STAGING_FILE ]; then exit 1; fi

              # Encrypt if key exists
              if [ -f /etc/backup/passphrase ]; then
                apk add --no-cache gnupg > /dev/null 2>&1
                cat $STAGING_FILE | gpg --batch --yes \
                  --passphrase-file /etc/backup/passphrase \
                  --symmetric --cipher-algo AES256 \
                  --output ${STAGING_FILE}.gpg
                mv ${STAGING_FILE}.gpg ${READY_DIR}.gpg
                rm $STAGING_FILE
              else
                mv $STAGING_FILE $READY_DIR
              fi

              find $READY_DIR -name "*.dump*" -mtime +7 -delete
            env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: password
            volumeMounts:
            - name: backup-storage
              mountPath: /backup
            - name: encryption-key
              mountPath: /etc/backup
              readOnly: true
          volumes:
          - name: backup-storage
            persistentVolumeClaim:
              claimName: backup-pvc
          - name: encryption-key
            secret:
              secretName: backup-encryption-secret
              optional: true
          restartPolicy: OnFailure

Apply:

kubectl apply -f backups/02-postgres-backup.yaml

🧩 Step 4: Job 2 — Redis Backup

Non-blocking backup (BGSAVE).

File: backups/03-redis-backup.yaml

apiVersion: batch/v1
kind: CronJob
metadata:
  name: redis-backup
  namespace: backup
spec:
  schedule: "0 2 * * *" 
  successfulJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: redis-client
            image: redis:8.0-alpine
            command:
            - /bin/sh
            - -c
            - |
              set -e
              TIMESTAMP=$(date +%Y%m%d_%H%M%S)
              STAGING_DIR="/backup/staging/redis"
              READY_DIR="/backup/ready/redis"
              mkdir -p $STAGING_DIR $READY_DIR

              redis-cli -h redis.prod.svc.cluster.local BGSAVE
              LAST_SAVE=$(redis-cli -h redis.prod.svc.cluster.local LASTSAVE)
              while [ "$(redis-cli -h redis.prod.svc.cluster.local LASTSAVE)" -le "$LAST_SAVE" ]; do sleep 2; done

              redis-cli -h redis.prod.svc.cluster.local --rdb - > ${STAGING_DIR}/redis_${TIMESTAMP}.rdb
              if [ -s ${STAGING_DIR}/redis_${TIMESTAMP}.rdb ]; then
                 mv ${STAGING_DIR}/redis_${TIMESTAMP}.rdb ${READY_DIR}/
                 find $READY_DIR -name "*.rdb" -mtime +7 -delete
              fi
            volumeMounts:
            - name: backup-storage
              mountPath: /backup
          volumes:
          - name: backup-storage
            persistentVolumeClaim:
              claimName: backup-pvc
          restartPolicy: OnFailure

Apply:

kubectl apply -f backups/03-redis-backup.yaml

🧩 Step 5: Job 3 — N8N Backup (Direct Upload)

Runs in prod, uploads to mega:k8s-backups/n8n.

File: backups/04-n8n-backup.yaml

apiVersion: batch/v1
kind: CronJob
metadata:
  name: n8n-backup
  namespace: prod
spec:
  schedule: "15 2 * * *" 
  successfulJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: n8n-client
            image: rclone/rclone:latest
            command:
            - /bin/sh
            - -c
            - |
              set -e
              TIMESTAMP=$(date +%Y%m%d_%H%M%S)
              # Copy config to temp to prevent read-only errors
              cat /config/rclone.conf > /tmp/rclone.conf

              STAGING_DIR="/tmp/staging/n8n"
              READY_DIR="/tmp/ready/n8n"
              mkdir -p $STAGING_DIR $READY_DIR
              FILENAME="n8n_data_${TIMESTAMP}.tar.gz"

              tar -czf ${STAGING_DIR}/${FILENAME} -C /n8n_data .

              if [ -s ${STAGING_DIR}/${FILENAME} ]; then
                mv ${STAGING_DIR}/${FILENAME} $READY_DIR
                rclone copy $READY_DIR mega:k8s-backups/n8n \
                  --config=/tmp/rclone.conf
                rclone delete mega:k8s-backups/n8n --min-age 15d \
                  --config=/tmp/rclone.conf
              fi
            volumeMounts:
            - name: n8n-pvc
              mountPath: /n8n_data
            - name: rclone-config
              mountPath: /config
              readOnly: true
          volumes:
          - name: n8n-pvc
            persistentVolumeClaim:
              claimName: n8n-pvc
          - name: rclone-config
            secret:
              secretName: rclone-secret
          restartPolicy: OnFailure

Apply:

kubectl apply -f backups/04-n8n-backup.yaml

🧩 Step 6: Job 4 — Mega.nz Sync (Central)

Uploads postgres and redis from local PVC to mega:k8s-backups.

File: backups/05-mega-sync.yaml

apiVersion: batch/v1
kind: CronJob
metadata:
  name: mega-sync
  namespace: backup
spec:
  schedule: "30 2 * * *" 
  successfulJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: rclone-sync
            image: rclone/rclone:latest
            command:
            - /bin/sh
            - -c
            - |
              # Copy config to temp to prevent read-only errors
              cat /secret/rclone.conf > /tmp/rclone.conf

              rclone sync /backup/ready mega:k8s-backups \
                --config=/tmp/rclone.conf \
                --transfers 4

              rclone delete mega:k8s-backups --min-age 15d \
                --config=/tmp/rclone.conf
            volumeMounts:
            - name: backup-storage
              mountPath: /backup
            - name: rclone-secret
              mountPath: /secret
              readOnly: true
          volumes:
          - name: backup-storage
            persistentVolumeClaim:
              claimName: backup-pvc
          - name: rclone-secret
            secret:
              secretName: rclone-secret
          restartPolicy: OnFailure

Apply:

kubectl apply -f backups/05-mega-sync.yaml

🧩 Step 7: Kubernetes Dashboard

1. Enable & Expose Service

microk8s enable dashboard
microk8s kubectl expose deployment kubernetes-dashboard -n kube-system \
  --name=kubernetes-dashboard --port=443 --target-port=8443

2. Ingress (Valid SSL)

Access at https://kube.kamleshmerugu.me.

File: k8s-stack/ingress/dashboard-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dashboard-ingress
  namespace: kube-system
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/proxy-ssl-verify: "false"
spec:
  ingressClassName: public
  tls:
    - hosts: [kube.kamleshmerugu.me]
      secretName: dashboard-tls-secret
  rules:
    - host: kube.kamleshmerugu.me
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: kubernetes-dashboard
                port:
                  number: 443

Apply:

kubectl apply -f k8s-stack/ingress/dashboard-ingress.yaml

3. Get Token

microk8s kubectl describe secret -n kube-system microk8s-dashboard-token

🧩 Step 8: Verification

Test the jobs immediately.

# 1. Run backups
kubectl create job --from=cronjob/postgres-backup manual-pg -n backup
kubectl create job --from=cronjob/redis-backup manual-redis -n backup
kubectl create job --from=cronjob/n8n-backup manual-n8n -n prod
kubectl create job --from=cronjob/mega-sync manual-sync -n backup

# 2. Check Mega.nz
# You should see folder: k8s-backups
# Inside: postgres/, redis/, n8n/

🧩 Step 9: Restore Procedures

Restore PostgreSQL

# 1. Download
kubectl run pg-download --image=rclone/rclone:latest --rm -it -n backup --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"d","image":"rclone/rclone:latest","command":["sh","-c","sleep 3600"],"volumeMounts":[{"name":"v","mountPath":"/b"},{"name":"c","mountPath":"/s","readOnly":true}]}],"volumes":[{"name":"v","persistentVolumeClaim":{"claimName":"backup-pvc"}},{"name":"c","secret":{"secretName":"rclone-secret"}}]}}' -- sh

# Inside shell:
rclone copy mega:k8s-backups /b/ready --config=/s/rclone.conf
exit

# 2. Restore
kubectl run pg-restore --image=postgres:17-alpine --rm -it -n backup --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"r","image":"postgres:17-alpine","command":["sh"],"env":[{"name":"PGPASSWORD","valueFrom":{"secretKeyRef":{"name":"postgres-secret","namespace":"backup","key":"password"}}}],"volumeMounts":[{"name":"v","mountPath":"/d"}]}],"volumes":[{"name":"v","persistentVolumeClaim":{"claimName":"backup-pvc"}}]}}' -- sh

# Inside shell:
gpg --batch --passphrase "YOUR_PASS" \
  --decrypt /d/ready/postgres/n8n_db_DATE.dump.gpg \
  --output /d/ready/restore.dump
psql -h postgres.prod.svc.cluster.local -U n8n -d postgres -c "DROP DATABASE n8n;"
psql -h postgres.prod.svc.cluster.local -U n8n -d postgres -c "CREATE DATABASE n8n;"
pg_restore -h postgres.prod.svc.cluster.local -U n8n -d n8n /d/ready/restore.dump

Restore Redis

# 1. Pull to local from backup PVC
kubectl run redis-helper -n backup --image=busybox --rm -i --restart=Never -- cat /backup/ready/redis/redis_DATE.rdb > /tmp/redis.rdb

# 2. Push to Redis Pod
REDIS_POD=$(kubectl get pod -n prod -l app=redis -o jsonpath='{.items[0].metadata.name}')
kubectl cp /tmp/redis.rdb ${REDIS_POD}:/data/dump.rdb -n prod

# 3. Restart
kubectl delete pod -n prod -l app=redis

Restore N8N

kubectl run n8n-restore --image=rclone/rclone:latest --rm -it -n prod --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"r","image":"rclone/rclone:latest","command":["sh","-c","sleep 3600"],"volumeMounts":[{"name":"n","mountPath":"/n8n_data"},{"name":"c","mountPath":"/s","readOnly":true}]}],"volumes":[{"name":"n","persistentVolumeClaim":{"claimName":"n8n-pvc"}},{"name":"c","secret":{"secretName":"rclone-secret"}}]}}' -- sh

# Inside shell:
rclone copy mega:k8s-backups/n8n /tmp/r --config=/s/rclone.conf
tar -xzf /tmp/r/n8n_data_DATE.tar.gz -C /tmp/r
mv /n8n_data/.n8n /n8n_data/.n8n_old
cp -r /tmp/r/.n8n /n8n_data/
exit

kubectl delete pod -n prod -l app=n8n

🎉 Checklist

Component	Strategy
Postgres	Staging -> Encrypt -> Ready -> Sync
Redis	BGSAVE -> Ready -> Sync
N8N	Direct Upload to `k8s-backups/n8n`
Retention	7 Days Local, 15 Days Cloud
Dashboard	Accessible at `https://kube.kamleshmerugu.me`

You now have a bulletproof, production-ready backup system that is Simple, Modular, and RBAC-Free! 🛡️

🚀 Part 3 — Deploy Production Stack on MicroK8s

kamlesh merugu — Tue, 13 Jan 2026 14:23:58 +0000

In this part, we move from a basic MicroK8s cluster to a real production stack with:

PostgreSQL database
Redis cache
n8n workflow automation
HTTPS using Let’s Encrypt
Kubernetes Dashboard on its own domain

All deployed using proper namespaces and ingress rules.

✅ Prerequisites

From previous parts, you must already have:

✅ MicroK8s installed and running
✅ kubectl alias working
✅ DNS records created:

Domain	Points To
`n8n.domain.com`	Your server public IP
`kube.domain.com`	Your server public IP

🔌 Step 1 — Enable Required MicroK8s Add-ons

microk8s enable dns storage ingress cert-manager dashboard metrics-server

Wait for everything:

kubectl get pods -A

All pods should be Running.

🧱 Step 2 — Project Structure

mkdir -p k8s-project/{03-postgres,04-redis,05-n8n,07-ingress}
cd k8s-project

🧩 Step 3 — Create Production Namespace

kubectl create namespace prod

🔐 Step 4 — Create Secrets

PostgreSQL Password

kubectl create secret generic postgres-secret \
  --from-literal=password="$(openssl rand -base64 24)" \
  -n prod

n8n Encryption Key

kubectl create secret generic n8n-secret \
  --from-literal=encryption-key="$(openssl rand -base64 32)" \
  -n prod

🐘 Step 5 — Deploy PostgreSQL

📁 03-postgres/postgres.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pvc
  namespace: prod
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 10Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: prod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16
          env:
            - name: POSTGRES_DB
              value: n8n
            - name: POSTGRES_USER
              value: n8n
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: password
          ports:
            - containerPort: 5432
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: postgres-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: prod
spec:
  selector:
    app: postgres
  ports:
    - port: 5432

Apply:

kubectl apply -f 03-postgres/postgres.yaml

🔴 Step 6 — Deploy Redis

📁 04-redis/redis.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  namespace: prod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
        - name: redis
          image: redis:7-alpine
          ports:
            - containerPort: 6379
---
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: prod
spec:
  selector:
    app: redis
  ports:
    - port: 6379

Apply:

kubectl apply -f 04-redis/redis.yaml

🔁 Step 7 — Deploy n8n

📁 05-n8n/n8n.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: n8n-pvc
  namespace: prod
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: n8n
  namespace: prod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: n8n
  template:
    metadata:
      labels:
        app: n8n
    spec:
      containers:
        - name: n8n
          image: n8nio/n8n:latest
          env:
            - name: DB_TYPE
              value: postgresdb
            - name: DB_POSTGRESDB_HOST
              value: postgres
            - name: DB_POSTGRESDB_DATABASE
              value: n8n
            - name: DB_POSTGRESDB_USER
              value: n8n
            - name: DB_POSTGRESDB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: password
            - name: N8N_ENCRYPTION_KEY
              valueFrom:
                secretKeyRef:
                  name: n8n-secret
                  key: encryption-key
            - name: N8N_HOST
              value: n8n.domain.com
            - name: N8N_PROTOCOL
              value: https
            - name: WEBHOOK_URL
              value: https://n8n.domain.com/
          ports:
            - containerPort: 5678
          volumeMounts:
            - name: data
              mountPath: /home/node/.n8n
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: n8n-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: n8n
  namespace: prod
spec:
  selector:
    app: n8n
  ports:
    - port: 5678
      targetPort: 5678

Apply:

kubectl apply -f 05-n8n/n8n.yaml

🔐 Step 8 — Create Let’s Encrypt Issuer

📁 07-ingress/issuer.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: your-email@domain.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          ingress:
            class: public

Apply:

kubectl apply -f 07-ingress/issuer.yaml

🌍 Step 9 — Ingress for n8n

📁 07-ingress/n8n-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: n8n-ingress
  namespace: prod
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: public
  rules:
    - host: n8n.domain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: n8n
                port:
                  number: 5678
  tls:
    - hosts:
        - n8n.domain.com
      secretName: n8n-tls

Apply:

kubectl apply -f 07-ingress/n8n-ingress.yaml

📊 Step 10 — Ingress for Kubernetes Dashboard

📁 07-ingress/dashboard-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dashboard-ingress
  namespace: kube-system
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  ingressClassName: public
  rules:
    - host: kube.domain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: kubernetes-dashboard
                port:
                  number: 443
  tls:
    - hosts:
        - kube.domain.com
      secretName: kube-dashboard-tls

Apply:

kubectl apply -f 07-ingress/dashboard-ingress.yaml

✅ Step 11 — Verify Everything

Pods

kubectl get pods -A

Certificates

kubectl get certificate -A

Services

kubectl get svc -n prod
kubectl get svc -n kube-system | grep dashboard

🎉 Result

URL	What You Get
https://n8n.domain.com	n8n UI
https://kube.domain.com	Kubernetes Dashboard

🔜 Coming Next

👉 Part 4 — Centralized Secrets with Doppler + External Secrets Operator

👉 Part 5 — Observability (Prometheus, Grafana, Loki)

👉 Part 6 — Automated Backups & Disaster Recovery (Velero + DB dumps)

🧱 Part 1 — Production-Ready MicroK8s Installation on Debian 12 (with snapd)

kamlesh merugu — Tue, 13 Jan 2026 12:39:52 +0000

This guide sets up a secure, production-grade MicroK8s Kubernetes cluster on Debian 12, ready for deploying:

n8n workflows
Postgres & Redis
Observabilit y stack (Prometheus, Grafana, Loki)
Centralized secrets (Doppler, in Part 2)

⚠️ Follow every step carefully—skipping steps can break the cluster.

✅ Supported Systems

Debian 12 (Bookworm) LTS
VPS, VM, or bare-metal
Minimum: 2 vCPU, 4 GB RAM

🔧 Step 1 — Update & Prepare the Server

sudo apt update && sudo apt full-upgrade -y
sudo reboot

Reconnect after reboot. Install essential tools:

sudo apt install -y \
  curl wget git vim htop neofetch \
  apt-transport-https ca-certificates gnupg lsb-release

✅ System updated and ready for Kubernetes.

👤 Step 2 — Create a Deployment User (Optional but Recommended)

Kubernetes should never run as root.

sudo adduser deploy
sudo usermod -aG sudo deploy
echo "deploy ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/deploy
sudo chmod 440 /etc/sudoers.d/deploy
su - deploy

🔁 Step 3 — Disable Swap

sudo swapoff -a
sudo sed -i '/\sswap\s/s/^/#/' /etc/fstab
free -h

Swap should show 0B.

🌐 Step 4 — Kernel Modules & Networking

sudo tee /etc/modules-load.d/k8s.conf <<EOF
br_netfilter
EOF

sudo modprobe br_netfilter

sudo tee /etc/sysctl.d/99-k8s.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

sudo sysctl --system
sysctl net.ipv4.ip_forward

📦 Step 5 — Install snapd

MicroK8s is distributed as a Snap package.

sudo apt install -y snapd
sudo systemctl enable --now snapd
sudo systemctl enable --now snapd.socket

Verify snap:

snap version

🚀 Step 6 — Install Latest MicroK8s

sudo snap install microk8s --classic

Adjust the channel to the desired stable version.

🧠 Step 7 — Add Deploy User to MicroK8s Group

sudo usermod -aG microk8s $USER
sudo chown -f -R $USER ~/.kube

Important: Log out and back in, or use:

newgrp microk8s

🔹 Step 8 — Enable Core Addons

MicroK8s includes essential production addons:

microk8s status --wait-ready
microk8s enable dns storage ingress metrics-server

dns — service discovery

storage — dynamic persistent volumes

ingress — NGINX ingress controller

metrics-server — node & pod metrics

🧰 Step 9 — Setup kubectl for Deploy User

MicroK8s ships its own kubectl. To avoid typing microk8s.kubectl:

sudo snap alias microk8s.kubectl kubectl

Optional: export kubeconfig for scripts/CI:

mkdir -p $HOME/.kube
microk8s config > $HOME/.kube/config
chmod 600 $HOME/.kube/config
export KUBECONFIG=$HOME/.kube/config
echo 'export KUBECONFIG=$HOME/.kube/config' >> ~/.bashrc
source ~/.bashrc

🌍 Step 10 — Optional: Remote kubectl Access

microk8s config > kubeconfig.yaml
scp kubeconfig.yaml user@localmachine:~/.kube/config

Adjust server: IP if needed for external access.

➕ Step 11 — Add Worker Nodes (Optional)

microk8s add-node

Follow instructions on worker nodes. Verify:

kubectl get nodes

🔎 Step 12 — Verify Cluster Health

microk8s status --wait-ready
kubectl get nodes
kubectl get pods -A
kubectl cluster-info

All kube-system pods should be Running.

✅ What You’ve Achieved

MicroK8s installed via Snap on Debian 12
Deploy user configured with group membership
Kernel & networking configured for Kubernetes
Swap disabled, core addons enabled
kubectl working for non-root deploy user
Ready for n8n, Postgres, Redis, observability, and secrets

🚀 Next Steps (Part 2)

Setup centralized secrets with Doppler
Install External Secrets Operator
Prepare namespaces & secret sync for workloads

Part 2: Installing MicroK8s on Ubuntu 24.04+

kamlesh merugu — Tue, 13 Jan 2026 11:54:57 +0000

MicroK8s is a powerful yet lightweight Kubernetes distribution perfect for:

🚀 Developers testing Kubernetes locally
🧪 CI/CD environments
🏠 Homelabs
🛠️ Edge and IoT systems
🧩 Single-node or small clusters

This configuration is ideal for running MicroK8s with multiple workloads, monitoring tools, and CI/CD pipelines.

📋 What You'll Learn

In this guide, you will:

✨ Install MicroK8s
🔒 Configure user permissions
🧰 Enable essential add-ons
🧪 Verify the cluster is running
🎉 Deploy a test application
🔄 Ensure services autostart on boot

❓ Why MicroK8s?

MicroK8s is:

Lightweight 🪶 — Minimal resource footprint
Easy to install ⚡ — One command setup
Zero external dependencies 🎯 — Everything bundled
Production-ready 🔥 — Enterprise-grade
Beginner-friendly and pro-friendly 👨‍💻 — Great for learning and production

It includes core Kubernetes components and plug-and-play add-ons like DNS, Ingress, Helm, and the Dashboard.

📦 Install MicroK8s

Update your system

sudo apt update && sudo apt upgrade -y

Install MicroK8s via Snap

sudo snap install microk8s --classic

This will take a few minutes. MicroK8s installs the latest stable Kubernetes release.

Verify installation

sudo microk8s status --wait-ready

Expected output:

microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none

🎉 Kubernetes is up and running!

👥 Add Your User to the MicroK8s Group

MicroK8s creates its own Unix group to manage access. Add your user to avoid using sudo every time:

sudo usermod -a -G microk8s $USER

Apply group changes immediately without logging out:

newgrp microk8s

Now you can run microk8s commands without sudo!

💡 Pro Tip: Create a Kubectl Alias

Typing microk8s kubectl every time is tedious. Let's create an alias so you can just type kubectl.

Add the alias to your bash configuration:

echo 'alias kubectl="microk8s kubectl"' >> ~/.bashrc

Apply the changes to your current session:

source ~/.bashrc

Test it:

kubectl version --client

From this point forward, all commands in this guide will use kubectl for brevity, but remember it is running the MicroK8s version.

🔧 Enable Essential Kubernetes Add-ons

Recommended add-ons

Enable DNS for internal service discovery:

microk8s enable dns

Enable storage for persistent volumes (databases, file storage):

microk8s enable storage

Enable Ingress for external access (domain routing, SSL):

microk8s enable ingress

Optional but useful add-ons

Enable Helm 3 package manager:

microk8s enable helm3

Enable Kubernetes Dashboard (UI):

microk8s enable dashboard

Enable Metrics Server (for monitoring CPU/RAM usage):

microk8s enable metrics-server

Check status

microk8s status

You should see all enabled add-ons listed as active.

🧪 Test Your Kubernetes Cluster

Check node status

kubectl get nodes

Expected output:

NAME       STATUS   ROLES   AGE   VERSION
hostname   Ready    <none>  5m    v1.31.3

STATUS should be Ready ✅

Check system pods

kubectl get pods --all-namespaces

All components should show Running or Completed status:

NAMESPACE     NAME                                     READY   STATUS    RESTARTS   AGE
kube-system   calico-node-xxxxx                        1/1     Running   0          5m
kube-system   coredns-xxxxx                            1/1     Running   0          4m
ingress       nginx-ingress-microk8s-controller-xxxxx  1/1     Running   0          3m

If everything shows Running, your cluster is healthy! 🎉

🐳 Deploy a Test NGINX App

Let's deploy a simple NGINX workload to verify everything works end-to-end:

Create deployment

kubectl create deployment nginx --image=nginx

Expose the deployment

kubectl expose deployment nginx --port=80 --type=NodePort

Get service details

kubectl get svc nginx

You'll see output like:

NAME    TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
nginx   NodePort   10.152.183.45   <none>        80:31234/TCP   10s

Note the NodePort (e.g., 31234).

Access the application

Open your browser and visit:

http://your-server-ip:31234

If you see the NGINX welcome page → 🎉 Your Kubernetes cluster is working perfectly!

🔄 Ensuring MicroK8s Auto-Starts on Boot

Since MicroK8s is installed via Snap, systemctl cannot be used directly. Snap manages startup behavior automatically, but let's verify and ensure it's active.

✔️ Step 1: Check Service Status

sudo snap services microk8s

You should see:

Service                              Startup
microk8s.daemon-apiserver            enabled
microk8s.daemon-containerd           enabled
...

If all services show Startup: enabled, MicroK8s is set to autostart ✅

✔️ Step 2: Enable Autostart (If Needed)

If services show disabled, simply run:

sudo snap start --enable microk8s

This enables the entire application service and its internal daemons.

✔️ Step 3: Reboot & Validate

Reboot the server to test autostart:

sudo reboot

After logging back in, run:

microk8s status --wait-ready

If you see microk8s is running, then autostart is confirmed! 🎉

Verify your NGINX app survived the reboot

kubectl get pods

Your nginx pod should be running.

💾 Resource Usage Check

Check how much of your 8GB RAM MicroK8s is using:

free -h

Check disk usage (out of 120GB):

df -h

MicroK8s typically uses:

~500MB-1GB RAM at idle
~2-3GB storage for base installation

This leaves plenty of resources for your workloads! 🚀

🛠️ Troubleshooting Tools

Check cluster health (Generates a report)

sudo microk8s inspect

Restart MicroK8s

sudo microk8s stop
sudo microk8s start

Check specific logs

# Check kubelet logs
journalctl -u snap.microk8s.daemon-kubelet -f

# Check API server logs
journalctl -u snap.microk8s.daemon-apiserver -f

📋 Summary Checklist

Step	Purpose	Status
✅ Install MicroK8s	Kubernetes runtime	Done
✅ Add user to group	Non-root access	Done
✅ Set up kubectl alias	Simplify commands	Done
✅ Enable add-ons	DNS, storage, ingress	Done
✅ Test with kubectl	Validate cluster	Done
✅ Deploy NGINX	Confirm workload scheduling	Done
✅ Enable autostart	Ensure services start on boot	Done

🧹 Clean Up Test Deployment (Optional)

Once you've verified everything works, you can remove the test NGINX deployment:

kubectl delete deployment nginx
kubectl delete service nginx

🎯 Final Thoughts

Congratulations! 🎉 Your 4-core, 8GB RAM, 120GB Ubuntu server is now running a fully functional MicroK8s cluster!

You have a robust foundation ready for production workloads.

🔙 Previous Guide

👈 Back to Part 1: Prerequisites & Basic Ubuntu Setup

🔜 What's Next?

Ready to deploy production applications? 🚀

👉 Continue to Part 3: Production Namespace, App Deployment & Configuration

In Part 3, you'll learn how to:

🏗️ Create production namespaces for workload isolation
🚀 Deploy PostgreSQL and n8n with proper configurations
📦 Set up Persistent Volume Claims for databases
🔐 Secure sensitive data using Kubernetes Secrets

🏗️ Kubernetes Automation Stack (n8n + Postgres + Redis + Observability + Centralized Secrets)

kamlesh merugu — Tue, 13 Jan 2026 11:21:01 +0000

📌 Project Overview

This project sets up a production-ready MicroK8s Kubernetes environment for hosting:

n8n — workflow automation platform
Postgres — relational database for n8n
Redis — cache & queue backend for n8n
Observability Stack — Prometheus + Grafana + Loki
Centralized Secrets — Bitwarden (SaaS) as single source of truth

Backup & restore is included for all critical services to ensure disaster recovery.

🔑 Project Goals

Centralized Secrets Management

Secrets live in Bitwarden only.
Auto-sync into Kubernetes via External Secrets Operator (ESO).

Clean, Modular Deployment

Each service has its own namespace & manifest/Helm structure.
Independent lifecycle per service.

Observability & Health Checks

Metrics for cluster nodes, Postgres, Redis, n8n workflows.
Alerts & dashboards via Grafana.

Production-Ready Infrastructure

Persistent storage (PV/PVC) for Postgres + Redis.
Ingress with HTTPS (cert-manager / Let’s Encrypt).
Scalable and maintainable architecture.

Backup & Restore Strategy

Backup Postgres, Redis, and n8n workflows.
Cluster-level snapshot for disaster recovery.
Easy restore procedure for production incidents.

🏛️ Architecture Diagram

                 ┌─────────────────────────┐
                 │ MicroK8s Cluster        │
                 │  (Namespaces & PVs)     │
                 └─────────┬──────────────-┘
                           │
   ┌───────────────────────┼────────────────────────┐
   │                       │                        │
   ▼                       ▼                        ▼
┌─────────┐           ┌─────────┐             ┌──────────┐
│ n8n     │           │ Postgres│             │ Redis    │
└─────────┘           └─────────┘             └──────────┘
     │                     │                        │
     └─────────────┬───────┴───────────────┬────────┘
                   ▼                       ▼
             ┌────────────┐           ┌────────────┐
             │ Prometheus │           │ Grafana    │
             └────────────┘           └────────────┘

🛠️ Tech Stack

Component	Version / Tool	Purpose
Kubernetes	MicroK8s v1.28+	Lightweight production-grade K8s
Secrets	Bitwarden SaaS	Centralized secrets vault
Secrets Sync	External Secrets Operator	Sync secrets from Bitwarden → K8s
Database	PostgreSQL	Primary n8n DB
Cache	Redis	Queue & cache for n8n
Workflow	n8n	Automation engine
Observability	Prometheus + Grafana + Loki	Monitoring & dashboards
Ingress	NGINX + cert-manager	HTTPS routing & certificates
Backup	Velero / Custom CronJobs	Backup & restore critical components

🗂️ Project Structure

k8s-project/
├── 01-cluster-setup/           # MicroK8s setup scripts & configs
├── 02-secrets/                 # Bitwarden + ExternalSecrets configs
├── 03-postgres/                # Postgres manifests / Helm / PVs
├── 04-redis/                   # Redis manifests / Helm / PVs
├── 05-n8n/                     # n8n manifests / Helm
├── 06-observability/           # Prometheus, Grafana, Loki
├── 07-ingress/                 # Ingress & cert-manager setup
├── 08-backup-restore/          # Backup & restore scripts / Velero configs
└── README.md

🚀 Setup Plan

Step 1 — Prepare Cluster

Update Debian nodes, disable swap, configure networking.
Install MicroK8s (single or multi-node).
Enable addons: dns, storage, ingress, metrics-server.

Step 2 — Configure Centralized Secrets

Setup Bitwarden Vault for all sensitive keys.
Install External Secrets Operator.
Create ExternalSecret manifests to sync secrets into K8s namespaces.

Step 3 — Deploy Databases

Deploy Postgres with PVCs and secrets from Bitwarden.
Deploy Redis with PVCs and secrets from Bitwarden.

Step 4 — Deploy n8n

Deploy n8n using manifests or Helm.
Configure secrets via ESO.
Expose via Ingress with HTTPS.

Step 5 — Deploy Observability Stack

Prometheus, Grafana, Loki.
Collect metrics from:
- MicroK8s nodes
- n8n, Postgres, Redis
Setup dashboards & alerts.

Step 6 — Validate & Harden

Test app workflows, DB connectivity, and secret sync.
Apply network policies, resource limits, and backup schedules.

Step 7 — Backup & Restore Strategy

Postgres: Scheduled pg_dump + store in PVC or external S3 bucket.
Redis: Scheduled RDB/AOF snapshots.
n8n workflows: Export via n8n API cronjobs.
Cluster state: Use Velero for cluster snapshots (PV + resources).
Provide restore scripts to recover from disaster with minimal downtime.

🔑 End Goal

By the end of this project:

Fully functional MicroK8s cluster running n8n + Redis + Postgres + Observability stack.
All secrets centralized in Bitwarden.
Safe, repeatable deployments with Helm/manifests.
Scalable, production-ready architecture with monitoring, backups, and HTTPS ingress.
Disaster recovery plan via backup & restore strategy.

Link to part 1

🐋 Complete Podman Desktop + WSL2 Setup Guide - Replace Docker Desktop for FREE!

kamlesh merugu — Sat, 14 Jun 2025 01:15:15 +0000

🚀 Complete Tutorial: Accessing Podman Desktop Containers from Ubuntu WSL

Ever wished you could seamlessly manage your Podman containers from your Ubuntu WSL terminal while still enjoying the slick UI of Podman Desktop on Windows? 🤔 You're in luck! This tutorial will guide you through setting up this powerful integration, giving you the best of both worlds. 💻✨

This is super useful for developers who want to leverage the Linux environment of WSL for their container workflows, while keeping Podman Desktop as their central hub for visual management. Let's dive in! 🐳

📋 Prerequisites

Before we get started, make sure you have these essentials in place:

Podman Desktop installed and happily running on your Windows machine.
An Ubuntu WSL distribution already installed and configured.
A Podman Machine created in Podman Desktop. (This usually happens automatically during the initial setup. 👍)

💡 Overview

Podman Desktop is smart! It sets up a special WSL virtual machine called "Podman Machine" and configures the Windows Podman client to chat with it. However, it doesn't automatically extend this courtesy to your other WSL distributions (like our beloved Ubuntu). Our mission: to bridge this communication gap! 🌉

Step 1: Access Your Ubuntu WSL Distribution 🚪

First things first, let's jump into your Ubuntu WSL session. Open your favorite Windows terminal (Command Prompt, PowerShell, or Windows Terminal) and type:

wsl --distribution Ubuntu

If Ubuntu is your go-to default distribution, a simple wsl will do the trick! 😉

Step 2: Install Podman Remote Client 📥

Instead of a full-blown Podman installation from Ubuntu's repositories, we're going for the leaner, meaner podman-remote binary. This ensures you're always on the cutting edge with the latest features and best compatibility! ✂️

Download the Latest Podman Remote Binary

Always check the Podman releases page for the absolute latest version.

Then, download and install the binary (remember to replace v5.5.1 with the actual latest version you found!):

wget https://github.com/containers/podman/releases/download/v5.5.1/podman-remote-static-linux_amd64.tar.gz

Configure the Binary 🛠️

Let's make sure your system knows where to find podman-remote and give it a friendly alias. Edit your .zshrc (or .bashrc if you're using Bash):

echo 'export PATH="$PATH:/usr/local/bin"' >> ~/.zshrc
echo 'alias podman="podman-remote-static-linux_amd64"' >> ~/.zshrc
source ~/.zshrc

Step 3: Identify Your Podman Machine Configuration 🕵️‍♀️

Before we connect, we need to know which socket your Podman Desktop is currently using.

Find Available Sockets 🔗

List the available Podman sockets in your WSL distribution:

find /mnt/wsl/podman-sockets/ -name '*.sock'

You should see something like this:

/mnt/wsl/podman-sockets/podman-machine-default/podman-root.sock
/mnt/wsl/podman-sockets/podman-machine-default/podman-user.sock

Identify the Active Socket 👀

Now, open a new Command Prompt or PowerShell window on Windows (don't close your WSL terminal yet!). Check which connection Podman Desktop is using:

podman system connection list

Look for the connection marked true in the Default column. The URI will tell you if it's rootful or rootless:

Rootful: ssh://root@127.0.0.1:PORT/run/podman/podman.sock (most common, default)
Rootless: ssh://user@127.0.0.1:PORT/run/user/1000/podman/podman.sock ### Step 4: Configure the Podman Connection 🤝

Based on whether your Podman Desktop is using rootful or rootless mode, let's set up the connection!

For Rootful Podman (Default) 👑

If your Podman Desktop is in rootful mode (which is typically the default):

podman system connection add --default podman-machine-default-root unix:///mnt/wsl/podman-sockets/podman-machine-default/podman-root.sock

For Rootless Podman 🌱

If your Podman Desktop is using rootless mode:

podman system connection add --default podman-machine-default-user unix:///mnt/wsl/podman-sockets/podman-machine-default/podman-user.sock

Step 5: Configure User Permissions 🔒

Your WSL user needs the right permissions to talk to the Podman Machine socket. Add your user to the appropriate group and then exit your WSL session to apply the changes. This step is crucial! 🔑

sudo usermod --append --groups 10 $(whoami)
exit

Step 6: Test the Connection! ✅

It's showtime! Restart your WSL session and let's verify everything is working smoothly.

wsl

Verify Group Membership 👥

Double-check that your user is now in the correct group:

groups

You should see uucp in the list (this corresponds to group ID 10 for rootful access).

Verify Podman Connection 🌐

Confirm that the connection is properly configured:

podman system connection list

Test Podman Version 🧪

Now, the moment of truth! Verify that Podman in your WSL can communicate with the remote machine:

podman version

You should see both Client and Server versions, indicating a glorious, successful connection! 🎉

Test Container Operations 📦

Let's run a quick test by launching a simple container:

podman run quay.io/podman/hello
podman ps -a --last 1

Voilà! The container should magically appear in both your WSL terminal output and in Podman Desktop's container list. Synchronization at its finest! 👯‍♀️

Step 7: Switching Between Rootful and Rootless (Optional) 🔄

Sometimes you might need to flip between rootful and rootless modes. Here's how:

To Switch to Rootless Mode:

In Windows terminal:
```
podman machine set --rootful=false
```

In WSL:

podman system connection add --default podman-machine-default-user unix:///mnt/wsl/podman-sockets/podman-machine-default/podman-user.sock

To Switch to Rootful Mode:

In Windows terminal:
```
podman machine set --rootful=true
```

In WSL:

podman system connection add --default podman-machine-default-root unix:///mnt/wsl/podman-sockets/podman-machine-default/podman-root.sock

🐛 Troubleshooting Corner

Ran into a snag? Don't worry, we've got you covered with some common fixes!

Permission Denied Errors 🚫

If you're seeing Permission denied errors:

Ensure you're in the correct group:
```
groups | grep uucp
```
If not, add yourself to the group again and exit/restart:
```
sudo usermod --append --groups 10 $(whoami)
exit
```
Restart WSL completely:
```
wsl --shutdown
```

Socket Not Found 🔌

If the socket files seem to be playing hide-and-seek:

Ensure Podman Desktop is running. 🏃‍♀️
Ensure the Podman Machine is started in Podman Desktop.
Check if the machine name is different (it might not be podman-machine-default):
```
ls /mnt/wsl/podman-sockets/
```

Connection Refused 🛑

If you're getting Connection refused errors:

Verify Podman Desktop is running.
Check that the Podman Machine is started.
Try restarting the Podman Machine from Podman Desktop. 🔄

⚙️ Custom Configurations

A couple of notes for more unique setups:

Custom WSL Distribution 🐧

If you're using a WSL distribution other than Ubuntu, the group names for group ID 10 might be different. You can find the correct group name using:

getent group 10

Custom Podman Machine Name 🏷️

If you've been adventurous and created a custom Podman Machine with a different name, simply replace `podman-machine-default` in the socket paths with your machine's actual name.

➡️ Next Steps

Congratulations! 🎉 You've successfully configured your Ubuntu WSL distribution to communicate with Podman Desktop's container engine. Now, the world of containerized development is your oyster within WSL!

You can now:

Use all Podman commands directly from your WSL terminal.
Manage containers that appear seamlessly in both your WSL and Podman Desktop.
Start experimenting with container orchestration tools like Podman Compose.
Deeply integrate container workflows into your WSL development environment.

🌟 Summary

You've done it! You've successfully linked your Ubuntu WSL with Podman Desktop, allowing you to enjoy the power of the Podman CLI from within WSL while benefiting from Podman Desktop's excellent graphical interface.

Key takeaways to remember:

Always use podman-remote for the latest features and best compatibility.
Configure the correct socket path based on your Podman Desktop's rootful/rootless setting.
Proper group permissions are essential for socket access.
The magic part: Both WSL and Podman Desktop will now show the same containers and images! ✨

Happy containerizing! 🚀