DEV Community: david

Postgres 16 18: Why You Can't Just Swap the Image

david — Mon, 05 Oct 2026 12:44:42 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

PostgreSQL major version upgrades are not swap-the-image operations. You can't change postgres:16 to postgres:18 in a Deployment and expect it to work. The data directory format changes, system catalogs are incompatible, and extensions need to be rebuilt.

When CNPG (CloudNativePG) upgraded from PG16 to PG18, the procedure required a fresh PVC, a full dump and restore, and a role discovery bug that nearly locked me out of the Authelia database.

View the complete homelab infrastructure source on GitHub 🐙

Why You Can't Swap the Image

PostgreSQL stores data in a format specific to its major version. The PG_VERSION file in the data directory tells PostgreSQL which version created it:

cat /var/lib/postgresql/data/PG_VERSION
# → 16

When PostgreSQL 18 starts and finds PG_VERSION = 16, it refuses to start:

FATAL: data directory has wrong ownership
HINT: The data directory was initialized by PostgreSQL 16. 
      Upgrade by running pg_upgrade.

pg_upgrade is the official tool for in-place major version upgrades. It copies data files from the old format to the new format, rewriting system catalogs and tuple headers. It works well on bare-metal PostgreSQL where you have direct filesystem access.

On Kubernetes with CNPG, pg_upgrade is not the right approach because:

CNPG manages the data directory through its operator — manual modifications are reverted
The PVC is bound to the cluster definition — changing the PostgreSQL version in the CRD doesn't automatically upgrade the data
CNPG's recommended migration path is dump-and-restore, not in-place upgrade

The Migration Procedure

Step 1: Dump from PG16

# Exec into the CNPG pod
kubectl exec -n database -it postgres-authelia-0 -- bash

# Dump the database
pg_dump -U postgres -Fc authelia > /tmp/authelia.dump

# Copy the dump to a temporary location
kubectl cp database/postgres-authelia-0:/tmp/authelia.dump ./authelia.dump

The -Fc flag produces a custom-format dump that's compressed and can be restored with pg_restore. The dump includes all data, schemas, roles, and extensions.

Step 2: Create Fresh PG18 Cluster

# kubernetes/system/postgres/cluster.yml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: postgres-authelia
  namespace: database
spec:
  imageName: ghcr.io/cloudnative-pg/postgresql:16.4  # temporary — will be updated
  instances: 1
  storage:
    size: 2Gi
    storageClass: local-path

Wait — the image is still PG16. That's intentional. CNPG creates the cluster with PG16 first, then upgrades the image to PG18 after the data is restored. This ensures the PVC and the operator agree on the initial state.

Step 3: Restore into PG18

After the cluster is running with PG16, update the image to PG18:

spec:
  imageName: ghcr.io/cloudnative-pg/postgresql:18.4

CNPG detects the image change, creates a new pod with PG18, and the old PG16 pod is terminated. The data directory is still PG16 format, so the new pod fails to start — which is expected.

Now restore the dump into the fresh PG18 data directory:

# Port-forward to the CNPG service
kubectl port-forward -n database svc/postgres-authelia 5432:5432 &

# Restore the dump
pg_restore -U postgres -d authelia --clean --if-exists ./authelia.dump

pg_restore handles the format conversion — it reads PG16-format data and writes it in PG18 format. The --clean --if-exists flags drop existing objects before restoring, ensuring a clean state.

Step 4: Update Authelia

Update the Authelia deployment to use the new Postgres 18 connection string (same host, same port, same database — the connection string doesn't change):

# Verify Authelia connects to the new database
kubectl logs -n apps -l app=authelia --tail=20
# → "Successfully connected to PostgreSQL 18.4"

The Role Discovery Bug

During the restore, pg_restore reported:

pg_restore: error: could not open input file "/tmp/authelia.dump": No such file or directory

The dump file was at a different path than expected. The real problem: the restore was running from a pod that had a different filesystem layout than the dump pod.

After fixing the path, the restore completed but Authelia failed to start:

FATAL: role "oc_dw" does not exist

The oc_dw role was a leftover from the original CNPG cluster initialization — CNPG creates a default operator role that isn't visible in pg_dump output because it's a replication role, not a regular database role.

The fix: create the missing role before restoring:

CREATE ROLE oc_dw WITH REPLICATION LOGIN;

The lesson: CNPG creates internal roles (oc_dw, streaming_replica) that aren't included in pg_dump output. After a dump-and-restore migration, these roles must be recreated manually.

The Fresh PVC Requirement

The critical step that most tutorials skip: you need a fresh PVC. You can't restore a PG16 dump into a PG16 data directory and then expect PG18 to read it. The data directory must be empty — PG18 creates its own data directory format on first start, and pg_restore populates it.

In CNPG, this means:

Delete the existing PVC (data loss — you need the dump)
Let CNPG create a new PVC with the PG18 image
Restore the dump into the fresh cluster

The PVC deletion is the scary part. If the dump is corrupted or incomplete, the data is gone. The verification before deletion:

# Verify dump is complete
pg_restore --list ./authelia.dump | wc -l
# → Should show hundreds of objects (tables, sequences, functions)

# Verify dump integrity
pg_restore --verbose --no-owner --no-privileges --dry-run ./authelia.dump
# → Should complete without errors

What I'd Change

Use CNPG's Backup/Restore instead of manual dump. CNPG supports pg_basebackup and WAL archiving to S3. A CNPG backup includes the operator roles and can be restored directly without the oc_dw gotcha. The manual dump approach was chosen because the existing cluster wasn't configured for CNPG backups at the time of migration.
Test the migration on a non-production cluster first. The oc_dw role discovery happened during the Authelia migration — the only SSO for every service. If the restore had failed, every service would be unreachable.

PostgreSQL major version upgrades on Kubernetes are the same challenge as Azure Database for PostgreSQL Flexible Server upgrades: Azure handles the in-place upgrade automatically, but the same data directory format incompatibility exists. The difference is that Azure abstracts the dump-and-restore behind a API call, while Kubernetes requires you to do it manually. The underlying PostgreSQL constraint is identical: major versions are not backward-compatible at the storage layer.

The Linux Command Line* is worth having on the shelf for exactly this kind of migration - pg_dump, pg_restore, kubectl cp, and a dozen other shell tools chained together under time pressure go a lot smoother when the shell itself isn't also something you're learning in the moment.

Renovate in Kubernetes: OOM, Hashicorp Downloads, and the 6GB Container

david — Sun, 04 Oct 2026 11:30:39 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Renovate is the dependency update bot that keeps my k3s cluster current, and it's the "auto-update" half of the operating model that decides what gets merged automatically vs. what waits for a human. It runs as a Kubernetes CronJob every 2 hours, checks for new container images, Helm chart versions, and Terraform provider releases, and opens PRs on GitHub.

For two weeks, it was failing silently on every run. The CronJob reported "Completed." The GitHub commits showed no new PRs. The logs showed nothing — because Renovate exits cleanly on config validation errors, and the CronJob's successfulJobsHistoryLimit kept only the last successful run.

View the complete homelab infrastructure source on GitHub 🐙

Failure 1: Invalid Preset

The first failure was a config error. The renovate.json file referenced an invalid preset:

{
  "extends": ["config:base", ":enableHelpfulPre-commit"]
}

:enableHelpfulPre-commit is not a real Renovate preset. Renovate's config validation caught the error and exited with a zero exit code — because config validation errors are treated as "nothing to do," not "something broke."

The CronJob's exit code was 0. Kubernetes considered the job successful. No alert fired. Renovate did nothing on every run.

The fix: remove the invalid preset and replace the deprecated config:base with config:recommended.

{
  "extends": ["config:recommended"]
}

Failure 2: OOM at 512Mi

After fixing the config, Renovate started running — and immediately OOM'd.

The container limit was 512Mi. Renovate's Node.js runtime, plus the GitHub API client, plus the Terraform provider registry client, plus the container image metadata parser, consumed more than 512Mi during a full scan.

The symptom: the Pod restarted with OOMKilled status. Kubernetes restarted it. The next run OOM'd again. After 3 failures, the CronJob marked the job as failed — but failedJobsHistoryLimit: 3 meant only the last 3 failures were visible, and they were being garbage-collected faster than I checked.

kubectl get jobs -n apps -l app=renovate
# NAME              COMPLETIONS   DURATION   AGE
# renovate-28197    0/1           ...        2m
# renovate-28196    0/1           ...        2h
# renovate-28195    0/1           ...        4h

The fix: raise the memory limit to 1Gi and set NODE_OPTIONS to limit the V8 heap:

env:
  - name: NODE_OPTIONS
    value: "--max-old-space-size=896"
resources:
  limits:
    memory: 6Gi
  requests:
    memory: 1Gi

The NODE_OPTIONS limit of 896MB (leaving ~100MB for native code and runtime overhead) prevents V8 from consuming the entire container memory during garbage collection cycles. Without it, V8 allocates memory up to the container limit and gets OOMKilled before the next GC cycle.

Failure 3: Terraform Provider Downloads

The third failure was the most subtle. Renovate was consuming 6GB of memory during Terraform provider update checks. The reason: Renovate downloads Terraform provider binaries to verify version compatibility.

For each Terraform provider in the repo (routeros, proxmox, cloudflare, garage), Renovate:

Queries the Terraform Registry API for available versions
Downloads the provider binary for the current and latest versions
Compares binary compatibility (provider schema, API version)
Opens a PR if a newer version is available

Step 2 is the memory problem. Terraform provider binaries are 50-200MB compressed. With 4 providers, plus their dependencies, Renovate downloads ~500MB of provider binaries during a full scan. The decompressed binaries and their metadata consume 2-3x that in memory.

The fix was two-part:

Raise the memory limit to 6Gi — enough for the full scan including provider downloads
Limit concurrent requests to the Terraform Registry:

{
  "terraform": {
    "concurrentRequestLimit": 2
  }
}

With concurrentRequestLimit: 2, Renovate downloads at most 2 provider binaries simultaneously, reducing peak memory usage from ~6GB to ~3GB. The scan takes longer, but it completes within the 6Gi limit.

The Silent Failure Pattern

The common thread across all three failures: Renovate exited cleanly. The CronJob reported success. No alert fired. The only evidence of failure was the absence of new PRs on GitHub.

This is the silent failure pattern:

The application fails internally but exits with code 0
Kubernetes considers the job successful
The successfulJobsHistoryLimit preserves the "successful" job
Nobody checks because the job "succeeded"

The fix: add a post-run check that verifies Renovate actually did something:

# Post-run check: verify PRs were opened
- name: Verify Renovate activity
  script: |
    PR_COUNT=$(gh pr list --repo dwoitzik/homelab-infrastructure \
      --author "app/renovate" --state open --json number --jq 'length')
    if [ "$PR_COUNT" -eq 0 ] && [ "$(date +%u)" -le 5 ]; then
      echo "WARNING: No open Renovate PRs on a weekday"
      # Alert via Discord webhook
    fi

The Current Configuration

# kubernetes/apps/renovate/renovate.yml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: renovate
  namespace: apps
spec:
  schedule: "0 */2 * * *"  # every 2 hours
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: renovate
              image: ghcr.io/renovatebot/renovate:43.245.0
              env:
                - name: RENOVATE_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: renovate-token
                      key: github-pat
                - name: NODE_OPTIONS
                  value: "--max-old-space-size=896"
              resources:
                limits:
                  memory: 6Gi
                  cpu: 2000m
                requests:
                  memory: 1Gi
                  cpu: 100m
          restartPolicy: Never
      backoffLimit: 2

The restartPolicy: Never with backoffLimit: 2 means the CronJob retries twice on failure, then gives up. Combined with the memory limit and heap size cap, Renovate completes its scan within resource bounds.

The Lesson

Silent failures in CronJobs are the most dangerous failure mode in a Kubernetes cluster — the same "reports success but did nothing real" pattern as ArgoCD showing Synced/Healthy on an Application that never actually applied. The job "succeeds," the operator doesn't check, and the dependency update pipeline silently stops working. Weeks pass without updates, security patches don't apply, and nobody notices until a vulnerability is disclosed for a package that Renovate would have updated.

The fix isn't just more memory — it's observability on the pipeline itself. Every CronJob that performs a critical function should have a post-run verification that confirms it actually did its job, not just that it exited cleanly.

CronJob observability is the same problem in Azure DevOps: a pipeline that succeeds but produces no artifact is indistinguishable from a pipeline that didn't run. Azure Monitor's Pipeline Analytics tracks success rate, not "did the pipeline produce meaningful output." The fix in both cases is the same: add a verification step that checks for the expected outcome (PRs opened, artifacts published, deployments completed) and alerts on absence.

The Phoenix Project* tells the same story from the human side of this exact failure mode - the automated system that everyone assumes is working, until someone finally checks.

Longhorn to NFS: Why Distributed Storage Didn't Make Sense Here

david — Mon, 28 Sep 2026 13:44:58 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Longhorn is designed for multi-node Kubernetes clusters. It replicates PersistentVolume data across nodes, so if one node dies, the data survives on the other two. It's a great distributed storage system.

I run three k3s nodes on a single physical host. All three nodes share the same NVMe. When Longhorn replicates a PVC across three nodes, all three replicas live on the same physical disk. There is no distribution. There is no redundancy. And when a node failed, Longhorn's own high-availability logic created a problem that wouldn't exist without it.

View the complete homelab infrastructure source on GitHub 🐙

The Multi-Attach Error

When a k3s node becomes unreachable (OomKill, kubelet crash, network blip), Longhorn marks its replicas as "rebuilding" and tries to attach the volume to a healthy node. But Longhorn's RWO (ReadWriteOnce) volumes can only be attached to one node at a time.

The sequence:

k3s-12 becomes unreachable (kubelet lost lease)
Longhorn detaches the PVC from k3s-12
Longhorn tries to attach the PVC to k3s-13
k3s-12 recovers and comes back online
Longhorn sees both nodes claiming the volume
Multi-Attach error: the volume is "attached" to two nodes simultaneously

The pod that needs the volume (Postgres, for example) can't start on either node because Longhorn refuses to mount a volume that's in a Multi-Attach state. The fix requires manually detaching the volume from both nodes and letting Longhorn re-attach it cleanly — a manual step during an outage, exactly when automation should be working.

On a multi-host cluster, this is a real HA scenario — the volume legitimately needs to failover to a different physical disk. On a single-host cluster, the "failover" is to the same NVMe, and the Multi-Attach error is pure overhead.

The NFS Alternative

NFS (Network File System) doesn't have a Multi-Attach problem because it's not a block storage system. NFS exports a directory over the network. Any number of clients can mount it simultaneously. There's no "attachment" state to conflict.

The NFS server runs on a dedicated LXC (ct-srv-nfs-01) with a ZFS-backed dataset:

# terraform/stacks/proxmox/lxc.tf
resource "proxmox_virtual_machine" "ct_srv_nfs_01" {
  vm_id   = 220
  name    = "ct-srv-nfs-01"
  memory { dedicated = 2048 }
  # ...
}

The k3s NFS provisioner creates PVCs on this NFS server:

# kubernetes/system/nfs-provisioner/application.yml
apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    helm:
      values:
        nfs:
          server: 10.0.20.100
          path: /archive
        storageClass:
          name: nfs-client
          defaultClass: true

When k3s-12 goes down, the pods reschedule to k3s-11 or k3s-13, mount the same NFS export, and continue where they left off. No Multi-Attach error, no manual detachment, no volume attachment state machine. NFS handles concurrent mounts natively.

The Trade-offs

What NFS Gives Up

Feature	Longhorn	NFS
Replication across nodes	Yes (3x by default)	No (single server)
Snapshots	Yes (per-volume)	Yes (ZFS snapshots)
ReadWriteMany	Yes	Yes
CSI driver	Yes	Yes (nfs-subdir-external-provisioner)
Performance	Block-level (faster)	Network-mounted (slower)
Node failure tolerance	Yes	No (single server)

NFS gives up replication and node-failure tolerance. If the NFS server dies, every PVC it serves is unavailable. On a single physical host, this is the same risk as Longhorn — both storage systems live on the same NVMe. Longhorn's replication doesn't help when the underlying disk is the single point of failure.

What NFS Gives Back

No Multi-Attach errors. Pods reschedule freely without volume attachment conflicts.
Simpler failure modes. NFS is either up or down. No "rebuilding," "degraded," or "reverted" states.
ZFS snapshots for backup. PBS backs up the NFS server's ZFS dataset, giving point-in-time recovery without Longhorn's snapshot overhead.
Lower resource usage. Longhorn runs a per-node process (manager + driver) consuming ~200MB RAM per node. NFS is a single process on one LXC.

The Storage Decision Matrix

After migrating from Longhorn to NFS, I defined the rule:

Application	Storage Class	Reason
PostgreSQL (CNPG)	local-path	CNPG manages its own replication
Garage S3 metadata	local-path	SQLite file-locking (the NFS trap)
Mealie, Home Assistant	local-path	SQLite databases
Everything else	nfs-client	Default, survives pod rescheduling

The principle: embedded databases go on local-path (file-locking), everything else goes on NFS (survivability). PostgreSQL doesn't fit either category because CNPG manages its own PV independently.

The Migration

Moving PVCs from Longhorn to NFS required:

Deploy the NFS server LXC with the correct ZFS dataset
Install the nfs-subdir-external-provisioner Helm chart
Copy data from Longhorn volumes to NFS (using kubectl cp or a temporary pod)
Update each application's storageClassName from longhorn to nfs-client
Delete the old Longhorn PVC (data is already on NFS)
Remove Longhorn from the cluster

Step 5 is the nerve-wracking part — deleting a PVC that contains production data. The verification before deletion:

# Verify data exists on NFS
kubectl exec -it temp-pod -- ls -la /mnt/nfs/authelia-data/
# → Confirm database files, config, etc.

# Verify application starts with NFS PVC
kubectl delete pod authelia-xxxxx  # force reschedule
# → Pod starts, mounts NFS, passes health checks

After the migration, Longhorn was uninstalled. The cluster went from three storage replicas on one disk to a single NFS server on the same disk — simpler, more predictable, and without the Multi-Attach false alarm.

Distributed storage on a single host is the same anti-pattern as Azure Zone-Redundant Storage across availability zones that share the same power source. If the underlying infrastructure isn't actually distributed, the replication layer adds complexity without adding resilience. The fix in both cases: match the storage topology to the actual infrastructure topology. Single host = single NFS server. Multi-host = distributed storage.

Designing Data-Intensive Applications* has the clearest explanation I've read of why replication only buys you resilience when the replicas are actually independent - which is the whole lesson of this migration in one sentence.

The Operating Model: What Should Auto-Update vs. What Needs a Human

david — Fri, 25 Sep 2026 11:00:27 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

A homelab that requires daily attention isn't a homelab — it's a job. The goal: a cluster that runs for weeks without intervention, alerts when something breaks, self-heals what it can, and waits for a human only when a human is actually needed.

This is the operating model that makes that work. Not aspirational — documented from what actually runs in production, including the deliberate gaps and the non-goals that keep the system honest.

View the complete homelab infrastructure source on GitHub 🐙

What Auto-Updates

Renovate runs every 2 hours via CronJob, tracking container images, Helm charts, and Terraform providers. But not everything gets the same treatment:

Auto-Merge (Patch/Minor/Digest, 3-Day Soak)

Stateless workloads that can be rolled back by ArgoCD's selfHeal if they break:

Dashboard widgets (Homepage, Uptime Kuma display)
Development tools (pre-commit hooks, linters)
Non-critical utilities (SearXNG, Mealie, MySpeed)

Renovate opens a PR, CI runs, and after 3 days of green the PR auto-merges. If the new version breaks, ArgoCD's syncPolicy.automated.selfHeal: true reverts to the previous version on the next sync.

PR-Only, Always (Manual Review Required)

Stateful and critical services where a bad upgrade can cause data loss or auth outage:

Databases: CNPG Postgres, Redis/Valkey
Auth: Authelia, Vault
Storage: Garage S3, Velero
Apps with data migrations: Nextcloud, Paperless, Gitea, Immich

Every update — patch, minor, or digest — gets a PR that requires manual review. The failure mode for these isn't "pod restarts" — it's "database schema mismatch" or "OIDC keys rotated."

PR-Only, Always (Major Version Bumps)

Every major version bump, any package, regardless of tier. Major versions break APIs, change configuration formats, and introduce migration steps that no automated tool can reliably handle.

Terraform: Never Auto-Applied

Every Terraform change goes through Atlantis: PR → plan → atlantis apply comment → apply. No exceptions. Terraform changes physical and VM state on a single-point-of-failure host — that always gets a human in the loop.

What Alerts (and Where)

Prometheus + Alertmanager route to Discord via webhook. ArgoCD's notifications-controller sends app-state events through a separate path.

Critical Alerts (Always Discord)

Everything with severity: critical:

Proxmox host temperature
k3s control-plane down
Storage >93% full
Postgres replication broken
Certificate expiry soon
Velero backup failed

Warning Alerts (Selective)

Only explicitly named warnings that matter at steady state:

KubePodCrashLooping — a pod is crash-looping
KubeJobFailed — a CronJob failed (covers Renovate itself)
KubeNodeNotReady — a node is unreachable
VeleroBackupPartialFailure — backup completed with warnings

Deliberately NOT Alerted

Routine warnings that would make Discord noisy without being actionable:

KubeMemoryQuotaOvercommit — LXC soft ceilings, not real pressure
KubeCPUThrottling — normal behavior for bursty workloads
KubePodNotReady during rolling updates — temporary, self-resolving

The principle: an alert nobody acts on trains people to ignore Discord. Every alert must have a corresponding action in the runbook. If there's no action, there's no alert.

What Self-Heals

ArgoCD: Automatic Sync + Self-Heal

All 41 live Applications have syncPolicy.automated.selfHeal: true. A merged manifest change deploys itself. Manual kubectl drift on anything ArgoCD tracks gets reverted automatically within minutes.

This is the primary self-healing mechanism. If someone manually patches a Deployment (during debugging, for example), ArgoCD reverts it on the next sync cycle. The cluster always converges to the git state.

Docker Containers: restart: unless-stopped

Media stack, Minecraft, AdGuard, Unbound — all run with restart: unless-stopped in Docker Compose. Crashes and host reboots self-recover the process. The data recovery is handled by NFS/ZFS, not Docker.

Kyverno: Audit-Only (Deliberately)

Kyverno runs in Audit mode — it reports policy violations in PolicyReports but does not block or auto-remediate. This is deliberate:

A bad policy in Enforce mode silently blocks legitimate deploys
On a single-operator homelab, there's nobody to notice the block quickly
Audit mode provides the visibility without the blast radius

The policies exist (require-resource-limits, disallow-latest-tag, disallow-privileged-containers) but they're watching, not enforcing. When I'm confident they won't cause false positives, they'll flip to Enforce.

What Stays Manual (On Purpose)

Terraform Apply

Always through Atlantis. A human comments atlantis apply. Never auto-applied, regardless of what changed. This is the one rule with no exceptions.

Stateful Service Bumps

Authelia, Vault, CNPG, Garage — reviewed before merge. The failure mode is data loss or auth outage, not "reroll the pod."

Snapshots Before State-Affecting Changes

Proxmox VM/CT snapshot or Velero PV snapshot, taken manually before applying anything that touches running state. The judgment of "is this change state-affecting" doesn't belong to a script.

Velero Restore Testing

Running, tested successfully at least once per namespace. But not yet exercised across every service. A backup that's only partly restore-tested is closer to hope than guarantee.

Full Cluster Rebuild

Documented in DISASTER-RECOVERY.md. Not automated. The target is fast, well-documented recovery — not zero-touch failover, because true HA isn't achievable on one physical host.

The Non-Goals

These are explicitly not targets:

Zero-downtime HA. Not achievable with one physical host. Not attempted.
Fully unattended major-version upgrades. Stateful service upgrades have repeatedly needed human judgment mid-migration (Postgres mount-point gotcha, capability-drop regression). Automating past that trades a 10-minute manual step for a much longer unattended-failure cleanup.
Alerting on everything. Deliberately tuned to steady-state-relevant signals only.

The Contract

This document is the contract between the infrastructure and the operator. Everything not listed here is assumed to be working. If it's not working and it's not in this document, it's a gap — not an expected manual task.

The operating model is a living document. As the cluster evolves (Cilium CNI, R2 offsite backup, Kyverno enforcement), the categories shift. But the principle stays: automate what's safe, alert what's important, and leave everything else to a human with context.

This operating model maps directly to enterprise SRE practices: SLO-based alerting replaces threshold alerting, runbooks define the human response for each alert type, and change management gates (like Atlantis's apply requirement) prevent unreviewed infrastructure changes. The difference is that enterprise environments have teams; a homelab has one person who needs to sleep through the night without Discord notifications.

The Phoenix Project* is the book that most shaped this document - it's fiction, but the underlying argument (know exactly what's automated, what's gated, and why) is the same one this operating model tries to make explicit instead of leaving implicit.

CPU Scheduling for etcd: Why Proxmox cpu.units Matters

david — Mon, 21 Sep 2026 12:35:09 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Update: this cluster has since moved off etcd entirely — a single-server k3s setup uses the
embedded SQLite datastore by default, and the multi-server HA path that would have used etcd
was never turned on here. The CPU-contention problem and the cpu.units fix below were real
while etcd was in play, and the underlying lesson (latency-sensitive, write-heavy workloads need
scheduling priority when they share a host with bursty ones like LLM inference or game servers)
still holds for anyone running real multi-server etcd today. Left as a historical/general
reference rather than a description of this cluster's current architecture.

etcd is the brain of Kubernetes. Every API call, every configmap update, every pod scheduling decision goes through etcd. When etcd is slow, everything is slow. When etcd times out, the API server becomes unreachable.

On a single Proxmox host running k3s VMs alongside Ollama LLM inference, Minecraft game servers, and Docker media workloads, etcd shares CPU with everything else. Without scheduling priority, a Minecraft player join and an Ollama model load can delay etcd's fdatasync calls enough to trigger leader-election timeouts.

The fix: cpu.units = 2048 on k3s VMs, giving them 2x the CPU scheduling priority over every other workload on the host — the same VMs that get staggered boot ordering to prevent I/O storms in the first place.

View the complete homelab infrastructure source on GitHub 🐙

How Proxmox CPU Scheduling Works

Proxmox uses the Linux CFS (Completely Fair Scheduler) with added weight controls. Each VM gets a cpu.units value that determines its share of CPU time when multiple VMs compete for the same physical cores.

The default is units = 1024. When two VMs with equal units compete for a single core, they each get 50% of the CPU time. When one VM has units = 2048 and another has units = 1024, the first gets 2/3 and the second gets 1/3.

This is not CPU pinning (which restricts a VM to specific cores). It's CPU weighting (which determines priority when cores are shared). On a host with 16 threads and 12+ VMs/LXCs, most cores are shared between multiple workloads.

The etcd Problem

etcd's performance depends on write latency. Every key-value operation (lease renewal, configmap update, secret sync) requires an fdatasync to the WAL (Write-Ahead Log) on disk. etcd's leader-election timeout is 5 seconds — if the leader can't renew its lease within that window, controller-runtime terminates the process.

On a single NVMe shared by all VMs, etcd's fdatasync competes with:

Ollama reading model weights from disk (14GB qwen2.5 model)
Minecraft world writes from the DMZ game server
NFS serving PVC data to k3s pods
ZFS txg commits flushing dirty data

Under normal load, this is fine — NVMe IOPS are high enough to handle all of them. But when Ollama starts loading a 26B model (Gemma 4) and Minecraft generates terrain simultaneously, the NVMe queue depth spikes, fdatasync latency increases, and etcd's lease renewal window shrinks.

Without CPU priority, etcd's fdatasync also competes for CPU time with these workloads. The kernel's CFS scheduler doesn't know that etcd's fdatasync is more important than Ollama's matrix multiplication — it just sees two processes requesting CPU time and allocates it equally.

The Fix

# terraform/stacks/proxmox/vm.tf

# k3s VMs — 2x scheduling priority
resource "proxmox_virtual_machine" "vm_srv_k3s_11" {
  cpu {
    cores = 4
    units = 2048  # 2x default
  }
}

# Ollama LXC — default priority
resource "proxmox_virtual_machine" "ct_srv_ai_01" {
  cpu {
    cores = 6
    units = 1024  # default
  }
}

# Minecraft LXC — default priority
resource "proxmox_virtual_machine" "ct_dmz_games_01" {
  cpu {
    cores = 2
    units = 1024  # default
  }
}

The k3s VMs get units = 2048. Everything else stays at the default 1024. When etcd and Ollama compete for the same CPU cycle, etcd gets 2/3 of the time and Ollama gets 1/3.

This doesn't reduce Ollama's throughput under normal conditions — when there's no contention, Ollama still gets 100% of the CPU it requests. The priority only kicks in when multiple workloads compete for the same cores simultaneously.

The Evidence

Before cpu.units = 2048, the CNPG operator was restarting due to leader-election timeouts:

kubectl get pods -n cnpg-system
# NAME                                    RESTARTS
# cnpg-cloudnative-pg-5f8b9c4d6-xk2p4   310

310 restarts in 21 days. Each restart's log showed:

Leader election retry deadline exceeded
context deadline exceeded

After applying cpu.units = 2048 and the corresponding leader-election timeout increase (--leader-renew-deadline=50), restarts dropped to zero.

The CPU priority alone didn't fix it — the leader-election timeout increase was also necessary. But the CPU priority reduced the frequency of fdatasync delays enough that the 50-second deadline is never challenged under normal load.

The RAM Priority Interaction

CPU priority and memory priority are separate in Proxmox. cpu.units affects CPU scheduling; memory.dedicated and memory.floating affect RAM allocation.

On this host, both matter:

etcd needs low-latency CPU for fdatasync (solved by cpu.units)
k3s control-plane needs guaranteed RAM for API server and scheduler (solved by memory.dedicated = 12284, the same VM-vs-LXC memory model covered in the overcommit guard writeup)

The RAM allocation is a hard reservation — 12 GB is always available for the k3s-11 VM. The CPU priority is a soft weighting — it only matters when cores are shared. Both are necessary: without RAM priority, the k3s VM could be ballooned down to 4 GB under pressure; without CPU priority, etcd could lose the fdatasync race to Ollama.

What I'd Change

Pin etcd to specific cores. CPU pinning would guarantee etcd always has CPU available, rather than just having priority. But pinning reduces overall CPU utilization — a pinned core can't be used by other VMs even when etcd is idle. On a 16-thread host with 12+ workloads, the utilization loss isn't worth it.
Monitor etcd fdatasync latency directly. Prometheus can scrape etcd's etcd_disk_wal_fsync_duration_seconds metric. An alert on p99 > 100ms would catch CPU contention before it triggers leader-election timeouts.

CPU scheduling priority is the same concept as Azure VM series selection: E-series VMs are memory-optimized, F-series are compute-optimized, and Dv5-series offer balanced resources. Choosing the wrong series for etcd (a latency-sensitive, write-heavy workload) produces the same performance degradation as running it without cpu.units on Proxmox. The difference is that Azure makes the choice at VM creation time, while Proxmox lets you adjust it dynamically.

Designing Data-Intensive Applications* is the best resource I know for understanding why a consensus system like etcd is so much more latency-sensitive than an ordinary stateless workload in the first place.

3-2-1 Backup in Practice: Velero + PBS + Offsite That Doesn't Work Yet

david — Mon, 21 Sep 2026 12:24:26 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

The 3-2-1 backup rule is simple: 3 copies of your data, on 2 different media types, with 1 copy offsite. My homelab implements all three layers. In theory. In practice, each layer has its own failure mode that I discovered only when I needed it — starting with the Garage S3 target Velero backs into, which is where layer 1 lives.

View the complete homelab infrastructure source on GitHub 🐙

The Three Layers

Layer 1: Velero → Garage S3 (in-cluster, daily 05:00 UTC)
    ↓
Layer 2: PBS → External HDD (local, daily 03:00 UTC)
    ↓
Layer 3: rclone → Google Drive (offsite, DISABLED)

Layer 1: Velero → Garage S3

Velero backs up k3s namespaces (apps, vault, database, argocd) to Garage S3 — a self-hosted S3-compatible object store running inside the same cluster.

# kubernetes/system/velero/schedule.yml
spec:
  schedule: "0 5 * * *"
  template:
    defaultVolumesToFsBackup: true
    includedNamespaces: ["apps", "vault", "database", "argocd"]
    excludedResources: ["events", "events.events.k8s.io"]
    ttl: 720h  # 30 days

defaultVolumesToFsBackup: true was the critical fix. Without it, Velero only captured Kubernetes manifests — not the actual PVC data. The backups "completed" for weeks with zero data.

After the fix, Kopia sidecars run alongside each pod, copying PVC contents to the Garage S3 velero bucket. Daily backups, 30-day retention.

The catch: Garage runs inside the cluster. If the cluster dies, both Velero and Garage are gone. Layer 1 is useful for recovering individual PVCs or namespaces within a running cluster — not for full cluster recovery.

Layer 2: PBS → External HDD

Proxmox Backup Server (PBS) backs up VM and LXC disk images to a Seagate 2TB External HDD* connected via USB.

# PBS backup job — runs daily at 03:00 UTC
# Backs up all VMs and LXCs to /mnt/backup (USB HDD)
proxmox-backup-client backup \
  --repository local:/mnt/backup \
  --ns homelab \
  vm/110/pct/200/pct/210/...

PBS handles deduplication, compression, and incremental backups. The USB HDD provides local, offline backup that survives cluster failures.

The gotcha: PBS and the k3s VMs share the same physical host. A host-level failure (PSU, NVMe death) takes out both the primary data and the PBS backup. Layer 2 is protection against software failure (corruption, accidental deletion), not hardware failure.

Layer 3: rclone → Google Drive (DISABLED)

The offsite layer was supposed to be rclone syncing from Garage S3 to Google Drive. In practice, Google's API throttled the sync to 1.6 KiB/s:

Transferred:   1.6 KiB / 4.2 GiB,  0.00%
Elapsed time:  2h 30m
Transfer rate: 1.6 KiB/s

Google Drive's API rate limiting for server-to-server transfers (no user interaction) is aggressive. For 4 GB of data, the sync would take days. Combined with Google's 750 GB/day upload limit for personal accounts, and the fact that the sync would need to run continuously to keep up with daily backups, the offsite layer was disabled.

The replacement (Cloudflare R2) is scaffolded but not active:

# kubernetes/system/velero/offsite-schedule.yml
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily-offsite
spec:
  schedule: "0 4 * * *"
  template:
    includedNamespaces: ["apps", "vault", "database", "argocd"]
    storageLocation: r2-offsite
    ttl: 168h  # 7 days

Waiting on a Cloudflare R2 account with real API credentials. R2 has no egress fees and no API rate limiting for the volume this backup produces (~4 GB/day). Activating it also fixes a second problem beyond throttling: Garage runs inside the same cluster Velero is protecting, so an in-cluster-only backup target is circular regardless of how fast it uploads.

The Restore Gotcha

The first time I tested a full restore, PBS hit an IP/MAC conflict:

Error: VM 211 is running on a different node (10.0.20.11)
Proxmox cannot start the VM because the MAC address is already in use

The issue: PBS restores the VM's network configuration exactly as it was, including the MAC address. If the original VM is still running (or its MAC is cached in the bridge), the restored VM can't start because Proxmox detects a MAC conflict on the virtual bridge.

The fix: restore to a temporary VMID, verify the restore is complete, then shut down the original and rename the restored VM. This is a manual multi-step process — not something you want to do under time pressure during a real disaster.

What Actually Works

Layer	Protects Against	Doesn't Protect Against
Velero → Garage	PVC corruption, namespace deletion, accidental kubectl delete	Cluster-wide failure (Garage is in-cluster)
PBS → USB HDD	Software corruption, accidental VM deletion, ZFS pool issues	Host hardware failure (PBS is on the same host)
rclone → Google Drive	Host hardware failure, theft, fire	Nothing yet — disabled due to throttling

The honest assessment: I have two functional backup layers, both on the same physical host. True offsite backup doesn't exist yet. The 3-2-1 rule is aspirational, not achieved.

The Verification Gap

The most important lesson: a backup you haven't restored is not a backup.

Velero backups run daily, PBS runs daily, and until recently neither had been fully restored. The Velero defaultVolumesToFsBackup gap existed for weeks because nobody tested a restore. The PBS IP conflict was discovered during the first full restore test.

After these discoveries, I added:

Monthly Velero restore test — restore the database namespace to a temporary namespace, verify Postgres starts and contains expected data
Quarterly PBS restore test — restore one VM to a temporary VMID, verify it boots and services are functional
Post-backup verification — velero backup describe --details | grep "Pod Volume Backups" after every scheduled backup

Backup verification is the same compliance requirement in Azure: ISO 27001 and NIS2 both require documented, tested restore procedures. Azure Backup reports "Completed" for VM snapshots, but the snapshot might not include the data disk if the backup policy was misconfigured. The only way to verify is to actually restore a test VM and confirm its contents — the same monthly drill I now run against Velero and PBS.

NFS vs local-path: The SQLite Trap That Corrupted My S3 Metadata

david — Mon, 14 Sep 2026 12:43:37 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Garage S3, my self-hosted S3-compatible object store, uses SQLite for its metadata database. One morning, Terraform state operations started failing with SQLITE_CORRUPT. The bucket metadata was gone. The terraform-state bucket, the Atlantis lock table, the Velero backup index — all of it stored in a SQLite file that was now corrupted.

The root cause: Garage was running on an NFS-backed PersistentVolume. NFS doesn't support the file-locking primitives that SQLite's WAL (Write-Ahead Logging) mode requires. Under concurrent access — Garage's metadata writer and a Velero backup reading the same database — the NFS lock delegation failed silently, and SQLite wrote to overlapping pages.

View the complete homelab infrastructure source on GitHub 🐙

The Storage Classes

My k3s cluster has two StorageClasses:

# NFS — for most workloads
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-client
provisioner: nfs-subdir-external-provisioner
parameters:
  server: 10.0.20.100
  path: /archive
reclaimPolicy: Retain

# Local-path — for embedded databases
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-path
provisioner: rancher.io/local-path
reclaimPolicy: Retain

NFS (nfs-client) is the default. It's backed by a dedicated LXC running an NFS server on ZFS, providing storage that survives pod rescheduling — a pod on k3s-12 can access the same PVC as a pod on k3s-13 because the NFS server is independent of any specific node.

Local-path (local-path) pins the PV to whichever node created it. If the pod reschedules to a different node, the PVC is inaccessible until the pod returns to the original node. This is a limitation, but it's the right trade-off for certain workloads.

Why SQLite and NFS Don't Work

SQLite's WAL mode requires fcntl() file locks — specifically, F_SETLK (non-blocking lock) and F_SETLKW (blocking lock). These locks coordinate access between concurrent processes writing to the same database file.

NFS handles file locks differently:

NFSv3: No native lock support. fcntl() calls return success but locks are local to the client — two NFS clients can both acquire an "exclusive" lock on the same file simultaneously.
NFSv4: Has LOCK operations, but the lock delegation model introduces latency and failure modes that SQLite's tight locking loop doesn't tolerate. If the NFS server is slow to respond to a lock request, SQLite's default 5-second busy timeout can expire, causing the application to retry — and the retry can conflict with the lock held by another client.
Kubernetes NFS provisioner: The nfs-subdir-external-provisioner uses NFSv4, but the lock delegation is handled by the NFS server's rpc.lockd daemon, which runs in a separate process space. Under concurrent load, lockd can lose track of which client holds which lock.

The result: SQLite thinks it has an exclusive lock, but another process (or the same process on a different connection) also has a lock. Both write to the database file. Pages overlap. The database corrupts.

The Garage Incident

Garage runs with two storage mounts:

# kubernetes/apps/garage/garage.yml
volumes:
  - name: data
    persistentVolumeClaim:
      claimName: garage-data      # NFS — bucket objects
  - name: meta
    persistentVolumeClaim:
      claimName: garage-meta      # local-path — SQLite metadata

The data volume (bucket objects) is on NFS — fine, because S3 object storage doesn't use file locks for individual files. The meta volume (SQLite database) was also on NFS initially. This worked until a Velero backup and a Terraform state write happened simultaneously.

Velero reads Garage's S3 API to enumerate backup objects. Terraform reads the same database to verify state file existence. Both hit SQLite through Garage's metadata layer. Under NFS, the concurrent reads triggered the lock delegation failure, and SQLite corrupted pages 169–184 of db.sqlite.

The Recovery

SQLite has a .recover command that can extract data from a corrupted database:

# Dump recoverable data from corrupted SQLite
sqlite3 db.sqlite ".recover" > recovered.sql

# Recreate the database from the dump
sqlite3 db_clean.sqlite < recovered.sql

The .recover command scanned every page of the corrupted file and extracted whatever data it could read. For pages 169–184 (the corrupted range), it found partial data — enough to reconstruct the bucket and key metadata, but not enough to guarantee referential integrity.

After recovery, the missing objects (terraform-state bucket and Atlantis lock key) had to be re-inserted manually via Python:

import sqlite3
import msgpack

conn = sqlite3.connect("db_clean.sqlite")
# Garage uses msgpack-encoded metadata with 'G2key'/'G2bkt' prefixes
# Re-insert the terraform-state bucket
conn.execute("INSERT INTO buckets (name, ...) VALUES (...)")
conn.commit()

The data was restored, but the trust was gone. The database could corrupt again under the same conditions.

The Fix

Move every embedded database to local-path:

# Garage — meta volume moved to local-path
volumes:
  - name: meta
    persistentVolumeClaim:
      claimName: garage-meta      # NOW: local-path (was: nfs-client)

The apps that need local-path:

App	Database	Why local-path
Garage S3	SQLite (metadata)	File-locking requirements
Mealie	SQLite (recipes)	WAL mode + concurrent access
Home Assistant	SQLite (state)	Inotify-based DB writes
Authelia	PostgreSQL (CNPG)	CNPG manages its own PV

PostgreSQL (via CNPG) doesn't have the SQLite lock problem because it uses its own file locking, but CNPG requires local-path or a CSI driver that supports ReadWriteOnce — NFS's ReadWriteMany semantics can confuse CNPG's WAL archiving.

The Trade-off

local-path means the PVC is pinned to one node. If the pod reschedules, it loses access to the data. For Garage, this is acceptable: Garage runs on a single node, and if that node goes down, the S3 data is unavailable regardless (it's on the same host).

For databases that need HA (Postgres, Redis), the solution isn't local-path or NFS — it's a managed operator (CNPG for Postgres) that handles replication and failover independently of the storage layer.

The principle: if the application uses file-level locking (SQLite, BoltDB, LMDB), it goes on local-path. If it uses network-level locking (PostgreSQL, MySQL), it goes on NFS or a managed operator.

SQLite on NFS is the same failure mode as running SQLite on an SMB share in a Windows domain: the file-locking semantics are fundamentally incompatible. In Azure, this maps to Azure Files (SMB-backed) vs. Azure Disk (block storage). Azure Files supports SMB locks but has the same delegation latency issues under concurrent access — any application that needs tight file-level locking should use Azure Disks, not Azure Files. The principle is identical: embedded databases need local, low-latency storage with native file-locking support.

Designing Data-Intensive Applications* covers exactly this class of correctness assumption — what a storage layer actually guarantees about concurrent access versus what an application silently assumes it guarantees — in far more depth than a corrupted db.sqlite file teaches you in the moment.

My Media Stack Lives in Two Containers and a Python CronJob

david — Mon, 14 Sep 2026 12:37:38 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

My media acquisition stack — SABnzbd for Usenet downloads, Sonarr for TV, Radarr for movies, Bazarr for subtitles, NZBHydra2 for indexer search — used to run as k3s Deployments. It worked, but three problems made it a bad fit for Kubernetes: GPU passthrough for Jellyfin transcoding, per-flow traffic isolation for indexer queries versus the actual download path, and the NFS file-locking trap for media libraries.

The solution: move the media stack out of k3s entirely. Jellyfin runs in its own GPU-passthrough LXC. The acquisition stack runs in a second LXC with Docker Compose. A Python CronJob in k3s bridges the two via Traefik Service+Endpoints.

Update: the first version of this stack wrapped the whole acquisition LXC in a Mullvad WireGuard tunnel via gluetun, on the assumption that every flow out of that box needed VPN protection equally. Two days later I tore that back out — see "The Traffic-Isolation Rethink" below for why a single blanket tunnel was the wrong model for what these five apps actually do.

View the complete homelab infrastructure source on GitHub 🐙

Why Kubernetes Was Wrong for Media

GPU Passthrough

Jellyfin needs GPU access for hardware video transcoding. The BMAX Mini PC* has an AMD Radeon Vega iGPU that supports VAAPI hardware transcoding. Proxmox GPU passthrough requires IOMMU group isolation — the GPU is passed to a single container or VM exclusively.

Kubernetes doesn't natively support GPU passthrough for LXCs. The nvidia-device-plugin works for NVIDIA GPUs on specific cloud providers, but for AMD iGPU passthrough on bare-metal Proxmox, you need a dedicated LXC with /dev/dri/renderD128 mapped directly.

Jellyfin runs in ct-srv-jellyfin-01 with:

# terraform/stacks/proxmox/lxc.tf
lxc_conf {
  desc = "Jellyfin - GPU passthrough"
  # GPU device mapped via pct set
}

k3s's own nodes here (vm-srv-k3s-11/12/13) are Proxmox VMs, not LXCs — a VM doesn't share the host kernel, so there's no "just bind-mount the device node" path the way there is for an LXC. Getting a k3s pod real GPU access on this hardware would mean classic VFIO passthrough of the iGPU to one specific VM: unbinding amdgpu from the Proxmox host and handing the whole device to vfio-pci instead. I looked at this seriously before ruling it out, because "GPU-in-Kubernetes" device plugins exist and I wanted to know if they'd apply.

They don't, for this hardware. AMD's own rocm/k8s-device-plugin targets ROCm compute (HIP/OpenCL) — a materially heavier stack than what VAAPI hardware transcoding actually needs, which is just /dev/dri visibility. And a Kubernetes device plugin can't manufacture GPU access a node's kernel doesn't already have — for a VM, that access only exists after real hypervisor-level VFIO passthrough, which a plugin doesn't do. Worse, this is a single consumer Ryzen APU, not a data-center part with SR-IOV or mediated-device support for splitting one GPU across VMs — passthrough would hand the entire GPU to exactly one of the three k3s VMs, and the Proxmox host itself (which uses amdgpu for its own display/telemetry) would permanently lose access to it. There's also no portability payoff to offset that cost: k3s's scheduler can't move a pod needing a passed-through device to a different node than the one VFIO was bound to, so the usual "GPU follows the pod" reason people move transcoding into Kubernetes never materializes on a single-host, single-iGPU homelab. It would just be Jellyfin running on a VM instead of an LXC, at the permanent cost of the GPU being unavailable to anything else on the host.

The LXC path avoids all of that: an LXC shares the host's kernel, so the host keeps the amdgpu driver bound and simply grants the container access to the resulting /dev/dri/renderD128 device node. Non-exclusive from the host's perspective, already proven working, no PCI device binding to get wrong. If this box ever gets a GPU with real SR-IOV support, this is worth revisiting — the constraint here is the specific hardware, not a principled objection to GPU workloads in Kubernetes.

The Traffic-Isolation Rethink

The acquisition LXC has two genuinely different outbound flows, and my first pass treated them as one problem. SABnzbd connects to Eweka (my Usenet provider) over NNTPS on port 563 — already encrypted end-to-end, and Usenet copyright enforcement works exclusively via BitTorrent peer-list monitoring, so there's no mechanism by which an ISP or rights-holder observes or reports Usenet downloads in the first place. NZBHydra2's indexer search queries are a completely different flow: plain HTTP/HTTPS lookups against third-party indexer sites, which does expose the home IP to whoever's on the other end, the same as browsing any site directly.

Wrapping the whole LXC in gluetun/Mullvad "solved" both at once, but it was the wrong tool for either: VPN on the download path halves throughput for no privacy benefit Eweka's own SSL doesn't already provide, and it risks Eweka flagging the account for apparent multi-subscriber IP sharing. I tore gluetun out two days after standing it up and replaced it with a model that actually matches the two flows:

SABnzbd → Eweka, direct, over SSL. No VPN. The ISP sees "connected to news.eweka.nl," never content — that's the whole job done by NNTPS alone.
NZBHydra2 → indexers, through a Tor SOCKS5 proxy, tor container at 172.28.1.10:9050, configured with no direct-connection fallback. Search queries are small and latency-tolerant, a good fit for Tor's limited bandwidth, and Tor is a better fit than a commercial VPN for metadata-only queries — no single operator to trust, no throughput to throttle. If Tor is down, indexer queries fail outright instead of silently leaking the home IP.

I verified this configuration directly rather than trusting the design intent on paper: sabnzbd.ini shows socks5_proxy_url = "" (empty — SABnzbd was never routed through Tor) with ssl = 1, ssl_verify = 2 confirming the direct-to-Eweka SSL path is real; nzbhydra.yml shows proxyType: SOCKS pointed at the Tor container with no fallback option enabled. One live exception I noticed and haven't chased down yet: one specific indexer bypasses Tor and connects directly (proxyIgnoreDomains) — possibly a site that blocks Tor exit nodes, possibly a leftover exception from before I understood this stack properly. Worth revisiting.

Tor's bandwidth genuinely can't handle bulk transfers, and routing downloads through it would be abusive to a network that exists for people who need anonymity for safety — never route the actual download path through Tor, only small metadata lookups.

The acquisition LXC (ct-srv-media-acq-01) runs all five apps this way — no blanket tunnel, no kill-switch sidecar to maintain, no shared failure mode between "is Eweka's SSL up" and "is the VPN provider's WireGuard endpoint reachable today."

NFS File Locking

Media libraries (downloaded files, metadata databases) live on NFS. As I wrote about in the SQLite trap article, NFS file-locking semantics don't work with embedded databases. Sonarr and Radarr use SQLite internally for their media databases — and those databases corrupt on NFS under concurrent access.

Moving the acquisition stack to a local LXC with local storage eliminates the NFS lock problem. The media files themselves (downloaded episodes, movies) still live on NFS for sharing, but the application databases stay local.

The Architecture

┌─────────────────────────────────────────────┐
│  k3s Cluster                                │
│  ┌─────────────────────────────────────┐    │
│  │ Python CronJob (every 10 min)       │    │
│  │ - Checks Sonarr/Radarr queue        │    │
│  │ - Clears stuck items                │    │
│  │ - Triggers Jellyfin library scan    │    │
│  └──────────┬──────────────────────────┘    │
│             │ HTTP via Traefik              │
│  ┌──────────▼──────────────────────────┐    │
│  │ Traefik IngressRoutes               │    │
│  │ - sabnzbd.woitzik.dev               │    │
│  │ - sonarr.woitzik.dev                │    │
│  │ - radarr.woitzik.dev                │    │
│  │ - bazarr.woitzik.dev                │    │
│  └─────────────────────────────────────┘    │
└──────────────────┬──────────────────────────┘
                   │ Traefik Service+Endpoints
┌──────────────────▼──────────────────────────┐
│  ct-srv-media-acq-01 (LXC, Tor for indexers)│
│  ┌─────────┐ ┌────────┐ ┌────────┐         │
│  │ SABnzbd │ │ Sonarr │ │ Radarr │         │
│  └─────────┘ └────────┘ └────────┘         │
│  ┌─────────┐ ┌──────────────┐              │
│  │ Bazarr  │ │ NZBHydra2    │              │
│  └─────────┘ └──────────────┘              │
└─────────────────────────────────────────────┘
┌─────────────────────────────────────────────┐
│  ct-srv-jellyfin-01 (LXC, GPU passthrough)  │
│  ┌─────────┐ ┌──────────────┐              │
│  │ Jellyfin│ │ /dev/dri/    │              │
│  │         │ │ renderD128   │              │
│  └─────────┘ └──────────────┘              │
└─────────────────────────────────────────────┘

The Traefik Bridge

k3s Traefik routes external traffic to services inside the cluster. But the media stack isn't in the cluster — it's in LXCs. The bridge: Traefik IngressRoutes point at Kubernetes Services, which use Endpoints objects with hardcoded IP addresses pointing at the LXC containers.

# kubernetes/apps/jellyfin/jellyfin.yml
apiVersion: v1
kind: Service
metadata:
  name: jellyfin
  namespace: apps
spec:
  ports:
    - port: 8096
      targetPort: 8096
---
apiVersion: v1
kind: Endpoints
metadata:
  name: jellyfin
  namespace: apps
subsets:
  - addresses:
      - ip: 10.0.20.254  # ct-srv-jellyfin-01
    ports:
      - port: 8096

This is the same pattern for all external services. The Service has no selector — it's a "headless" Service where the Endpoints are manually maintained. Traefik doesn't know or care that the backend is an LXC instead of a pod.

The IngressRoute for Jellyfin:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: jellyfin
  namespace: apps
spec:
  entryPoints: [websecure]
  routes:
    - match: Host(`media.woitzik.dev`)
      kind: Rule
      middlewares: [{name: authelia}]
      services:
        - name: jellyfin
          port: 8096
  tls:
    secretName: wildcard-woitzik-dev-tls

The media stack gets Authelia protection, wildcard TLS, and Cloudflare Tunnel external access — the same as every in-cluster service. The only difference is the backend IP.

The Python CronJob

A Python CronJob runs every 10 minutes inside k3s, bridging the gap between the media stack and the cluster:

# CronJob that monitors media acquisition
schedule: "*/10 * * * *"
jobTemplate:
  spec:
    template:
      spec:
        containers:
          - name: media-watchdog
            image: python:3.12-slim
            command:
              - python
              - /scripts/media-watchdog.py

The script:

Queries Sonarr/Radarr APIs for stuck queue items (downloads stuck for >30 minutes)
Clears stuck items and re-triggers the download
Checks Jellyfin's library scan status
Posts status to Discord via webhook

Without this CronJob, stuck downloads sit indefinitely. Sonarr and Radarr don't have built-in queue monitoring — they trust the download client to report status, and SABnzbd sometimes silently fails without notifying the *arr stack.

What I'd Change

Use a proper service mesh for cross-boundary traffic. The manual Endpoints pattern works but is fragile — if the LXC IP changes, the Endpoints must be updated manually. A DNS-based service discovery (Headscale DNS entries, for example) would be more resilient.
Move the CronJob to a native LXC cron. The Python script runs in a k3s Pod but talks to LXC-hosted services. It has no business being in the cluster. A systemd timer on the media acquisition LXC would be simpler.

The media stack's departure from Kubernetes is the same pattern as running stateful workloads outside AKS: GPU workloads go to dedicated VMs with GPU passthrough, traffic-sensitive workloads get network-level isolation instead of a sidecar, and file-locking workloads go to local SSDs. Kubernetes excels at stateless, horizontally-scalable workloads. Media transcoding and acquisition are neither — they're stateful, single-instance, and hardware-dependent. The right platform for them is the bare metal underneath, not the orchestration layer on top.

Private AKS on Azure: No Public API Server, No Public Node IPs, No Default Outbound

david — Mon, 07 Sep 2026 19:17:12 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Run az aks create with the defaults and you get a public API server FQDN, a Standard Load Balancer giving every node its own path to the internet, and — unless you go out of your way to stop it — an admin kubeconfig that works from anywhere with the right token. None of that is a bug. It's the fast path, and for a demo cluster it's fine.

For a cluster inside a governed Hub & Spoke, it's three separate audit findings. This is the Terraform to close all three at once: a private API server, forced-tunneled egress through a firewall you already control, and private-endpoint-only ACR and Key Vault.

Get the base template free on GitHub 🐙

Target Architecture

                         ┌─────────────────────┐
                         │   Azure Firewall     │  ← you supply the IP
                         │  (existing / NVA)     │
                         └──────────┬───────────┘
                                    │ UDR: 0.0.0.0/0
                         ┌──────────┴───────────┐
                         │   vnet-aks            │
                         │                       │
              ┌──────────┴──────────┐  ┌─────────┴──────────┐
              │  snet-aks-nodes     │  │ snet-private-       │
              │  (default-deny NSG) │  │ endpoints            │
              │                     │  │  - ACR (Premium)     │
              │  Private AKS        │  │  - Key Vault         │
              │  (no public FQDN)   │  │                      │
              └─────────────────────┘  └─────────────────────┘

Three engineering decisions drive this, same as the Hub & Spoke module this one is designed to sit behind:

No public control-plane endpoint — reachable only from inside the VNet, a peered network, or a VPN
Forced tunneling, not default outbound — every node packet leaves through a firewall you control
Private-endpoint-only dependencies — ACR and Key Vault never touch the public internet either

Step 1: The API Server Has No Public FQDN

resource "azurerm_kubernetes_cluster" "this" {
  # ...
  private_cluster_enabled             = true
  private_dns_zone_id                 = azurerm_private_dns_zone.aks.id
  private_cluster_public_fqdn_enabled = false

  network_profile {
    network_plugin = "azure"
    network_policy = "azure"
    outbound_type  = "userDefinedRouting"
  }
}

private_cluster_public_fqdn_enabled = false is the setting people miss. private_cluster_enabled = true alone still publishes a public FQDN that resolves to nothing reachable — a fingerprintable artifact for no benefit. Turning it off means az aks show doesn't leak a hostname at all.

The cluster needs its own Private DNS Zone for the API server (privatelink.<region>.azmk8s.io), linked to the VNet — same DINE-policy-safe lifecycle.ignore_changes pattern as the Hub & Spoke module's centralized zones, because Azure Policy tagging fights this resource too.

Step 2: `userDefinedRouting` — the Setting That Actually Matters

network_plugin = "azure" and network_policy = "azure" get most of the attention in AKS networking guides. The setting that actually determines whether traffic is inspectable is outbound_type.

Left at its default (loadBalancer), AKS provisions its own Standard Load Balancer outbound rule and every node gets a direct, un-inspected path to the internet — a private API server with fully public node egress is a common half-measure that looks locked down in the portal and isn't.

resource "azurerm_route_table" "aks_egress" {
  route {
    name                   = "default-via-firewall"
    address_prefix         = "0.0.0.0/0"
    next_hop_type          = "VirtualAppliance"
    next_hop_in_ip_address = var.firewall_private_ip
  }
}

resource "azurerm_subnet_route_table_association" "aks_nodes" {
  subnet_id      = azurerm_subnet.aks_nodes.id
  route_table_id = azurerm_route_table.aks_egress.id
}

outbound_type = "userDefinedRouting" tells AKS to trust this route table instead of provisioning its own path. The Route Table has to exist and be associated to the node subnet before the cluster is created — depends_on enforces the ordering, because AKS validates the UDR is actually in place at cluster-creation time and fails otherwise.

Step 3: ACR and Key Vault Without a Public Endpoint

Nodes still need to pull images and (often) read secrets. The naive fix — admin_enabled = true on the registry, a shared pull secret in a Kubernetes Secret — is a credential that outlives any single deployment and shows up in kubectl get secret -o yaml for anyone with namespace read access.

resource "azurerm_container_registry" "this" {
  sku                            = "Premium" # required for Private Link
  admin_enabled                  = false
  public_network_access_enabled  = false
}

resource "azurerm_role_assignment" "aks_acr_pull" {
  scope                            = azurerm_container_registry.this[0].id
  role_definition_name             = "AcrPull"
  principal_id                     = azurerm_kubernetes_cluster.this.kubelet_identity[0].object_id
  skip_service_principal_aad_check = true
}

The kubelet identity — the identity that actually pulls images, not the cluster's control-plane identity — gets AcrPull directly. No credential to rotate, nothing in a Secret, nothing that outlives the node it's bound to. Key Vault follows the identical shape: RBAC-authorized, public_network_access_enabled = false, kubelet identity gets Key Vault Secrets User, ready for the CSI Secrets Store driver to mount without ever touching a static key.

What This Doesn't Do

It's a starting cluster, not a finished platform. No Entra ID–integrated kubectl auth (add azure_active_directory_role_based_access_control yourself), no multi-pool topology, no cluster autoscaler wiring, sku_tier defaults to Free (no uptime SLA — set Standard for production). And it assumes you already have a firewall to hand it a private IP — pair it with the Azure Firewall Forced Tunneling module if you don't.

For the operational reality of running production Kubernetes past the "it deployed" stage — the workflows and failure modes that don't show up in a getting-started guide — GitOps and Kubernetes* is the reference I've actually kept open while doing this.

🐙 Private AKS - Zero-Trust Edition — free, MIT licensed

private_cluster_enabled with no public FQDN at all
Forced tunneling via userDefinedRouting - no default outbound path
Private-endpoint-only ACR (Premium) and Key Vault, kubelet identity RBAC only
Zero-Trust node subnet NSG, DINE-policy-safe Private DNS Zones
Full source code - no lock-in, no black box

Get it: GitHub

Full source code · free forever · no account, no checkout

Staggered VM Boot: How I Prevented a Load Average of 147

david — Mon, 07 Sep 2026 12:29:57 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

After a host reboot, every VM and LXC on the Proxmox host started simultaneously. Twelve containers and VMs, all booting at once, all hitting the same NVMe for root filesystem reads, service starts, and NFS mounts. The host load average hit 147.

For context: load average represents the number of processes in the run queue or waiting for I/O. On a 16-thread CPU, a load average of 147 means 147 processes are competing for CPU or disk time. Every service was slow to start, k3s took minutes to become ready, and DNS didn't resolve for the first 90 seconds because the Raspberry Pi DNS nodes were waiting for services that hadn't booted yet.

The fix wasn't more resources — it was boot order.

View the complete homelab infrastructure source on GitHub 🐙

The Boot Storm

When the Proxmox host starts, all VMs and LXCs configured with onboot=1 start simultaneously. The host's NVMe handles root filesystem reads for every container, plus the ZFS txg commits from the NFS server, plus etcd writes from the k3s control-plane.

The simultaneous startup creates a thundering herd: 12 processes all requesting I/O at the same time, the NVMe queue depth maxes out, I/O latency spikes, and services that depend on each other (k3s needs NFS, k3s apps need DNS, DNS needs k3s services) enter a cascading wait state.

The load average doesn't just spike and recover — it compounds. Services that fail to start within their timeout window retry, adding more processes to the queue. k3s control-plane tries to mount NFS volumes, NFS is slow because it's competing with 11 other containers for I/O, k3s retries the mount, adding more load.

The Fix: Staggered Boot Order

Proxmox supports startup order with delays. The configuration in Terraform:

# terraform/stacks/proxmox/lxc.tf

# NFS first — k3s depends on it
resource "proxmox_virtual_machine" "ct_srv_nfs_01" {
  # ...
  startup = {
    order = 1
    up    = 30  # wait 30s after boot before starting next
  }
}

# k3s control-plane — waits for NFS
resource "proxmox_virtual_machine" "vm_srv_k3s_11" {
  # ...
  startup = {
    order = 2
    up    = 30  # 30s after NFS is up
  }
}

# k3s workers — 30s apart from each other
resource "proxmox_virtual_machine" "vm_srv_k3s_12" {
  startup = {
    order = 3
    up    = 30
  }
}

resource "proxmox_virtual_machine" "vm_srv_k3s_13" {
  startup = {
    order = 4
    up    = 30
  }
}

The order:

NFS (order 1) — boots first, 30s head start. k3s PVCs mount from NFS, so NFS must be ready before k3s starts.
k3s-11 (order 2) — control-plane + etcd. Boots 30s after NFS. Needs NFS for system PVCs.
k3s-12 (order 3) — worker. Boots 30s after control-plane. Needs API server ready.
k3s-13 (order 4) — worker. Boots 30s after k3s-12.
LXCs (order 5+) — everything else. Docker workloads, media stack, DMZ.

Total boot sequence: ~3 minutes for everything to be online. Previously: all at once, 147 load average, 5+ minutes to stability.

Why NFS First

NFS is the foundation of the storage layer. Every k3s PVC (Authelia, Vaultwarden, Paperless, Nextcloud) mounts from the NFS server at 10.0.20.100. If NFS isn't ready when k3s starts, the pod mount attempts fail, Kubernetes retries with exponential backoff, and the pods sit in ContainerCreating for minutes.

NFS itself depends on ZFS — the NFS export directory lives on the ZFS pool. ZFS needs a few seconds after boot to complete any pending txg commits and mount the pool. The 30-second head start gives ZFS and NFS time to stabilize before k3s starts hammering them with mount requests.

The CPU Scheduling Priority

Beyond boot order, k3s VMs get CPU scheduling priority via cpu.units:

# k3s VMs get 2x scheduling priority
cpu {
  cores = 4
  units = 2048  # default is 1024
}

# LXCs stay at default priority
cpu {
  cores = 2
  units = 1024  # default
}

cpu.units tells the Proxmox scheduler how to weight CPU time when multiple VMs compete for the same physical cores. units=2048 means k3s VMs get twice the CPU scheduling priority over LXCs. When Ollama is running LLM inference on the AI LXC and k3s needs CPU for etcd fdatasync, etcd wins.

This matters during boot too: even with staggered starts, there's overlap between late-booting LXCs and already-running k3s workloads. The CPU priority ensures k3s gets scheduling preference.

The `onboot` Gotcha

Proxmox's bpg/proxmox Terraform provider doesn't reliably manage the onboot attribute. terraform plan always shows "No changes" regardless of the live value — a known limitation of the provider.

This means onboot must be set manually after any LXC recreate:

pct set <id> -onboot 1

I discovered this the hard way: after recreating a container via Terraform, onboot defaulted to 0. The next host reboot silently skipped that container. The k3s control-plane node came up without its NFS mount, and half the cluster was in CrashLoopBackOff until I noticed.

The workaround: a manual pct set step in the operations runbook, applied after every LXC creation. Not ideal, but documented and repeatable.

Before vs. After

Metric	Before (simultaneous)	After (staggered)
Peak load average	147	12
Time to k3s ready	5+ minutes	90 seconds
Time to DNS functional	90 seconds	30 seconds
I/O wait %	85%	15%
Failed mount attempts	12-15	0

The staggering didn't add total boot time — it redistributed the I/O load over 3 minutes instead of concentrating it in 30 seconds. Services come up later individually but the cluster as a whole is stable sooner because nothing is fighting for I/O. The same host also runs a memory overcommit guard for the RAM side of this problem — boot order fixes the I/O storm, the guard fixes the memory one.

Boot storm mitigation is the same problem in Azure: when you scale out a VMSS from 0 to 50 instances, all 50 hit the Azure fabric simultaneously. Azure handles this with staggered placement and shared disks, but the principle is identical — spread the I/O load over time instead of concentrating it. In a homelab, you do it yourself with startup.order and startup.up delays.

Designing Data-Intensive Applications* has a genuinely useful framing for this kind of thundering-herd problem, even though it's written about databases rather than hypervisors - the queueing math behind "everything wants the same resource at the same instant" doesn't care what the resource actually is.

The Overcommit Guard: How a Python Script Prevents Host Freezes

david — Thu, 03 Sep 2026 19:30:32 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

My Proxmox host froze three times in one week. The root cause was RAM pressure pushing ZFS into an I/O stall on a single shared NVMe. After fixing the freeze (ZFS tuning, cache=none+aio=native), I needed a way to prevent it from ever happening again — a CI guard that catches memory overcommit before it hits the host. It's the same host where uncontrolled boot storms once pushed load average to 147; memory pressure and I/O pressure are two versions of the same single-host bottleneck.

The first version of the guard was wrong. It summed VM and LXC memory allocations together against one ceiling, and flagged 85 GB as "allocated" on a host with 62 GB of physical RAM. A live check showed only 41 GB actually in use — 21 GB available. The guard was overstating pressure by 40 GB because it treated two different memory physics as the same thing.

The fix was a two-tier model that understands the difference between a VM reservation and an LXC ceiling.

View the complete homelab infrastructure source on GitHub 🐙

The Two Memory Physics

VM `dedicated` — Real Reservation

When you set dedicated = 12288 (12 GB) on a Proxmox VM, QEMU pre-allocates that memory as a host process. The moment the VM starts, 12 GB of physical RAM is gone — reserved for the QEMU process, not available for anything else. The host's free -h reflects this immediately.

This is a hard reservation. If the sum of all VM dedicated values plus ZFS ARC plus host overhead exceeds physical RAM, the host is overcommitted. The kernel will start swapping, ZFS will stall waiting for I/O, and the freeze pattern repeats.

LXC `dedicated` — Soft Ceiling

When you set dedicated = 4096 (4 GB) on a Proxmox LXC, you're setting memory.max in the cgroup. This is a ceiling the kernel enforces only if the container actually tries to use that much. It reserves nothing on the host up front.

A container with dedicated = 4096 might be using 800 MB. The host sees 800 MB, not 4 GB. The remaining 3.2 GB is available for other workloads. This is why the original guard was wrong: summing LXC dedicated values counts memory that isn't actually consumed.

A live check on the host confirmed this:

# Check actual LXC memory usage vs configured limits
for ct in $(pct list | awk 'NR>1 {print $1}'); do
  limit=$(pct config $ct | grep "dedicated" | awk '{print $2}')
  actual=$(pct exec $ct -- cat /sys/fs/cgroup/memory.current 2>/dev/null)
  echo "CT $ct: limit=${limit}MB actual=$((actual/1024/1024))MB"
done
# → Most CTs using 20-40% of their configured limit

The Two-Tier Guard

Hard Gate (CI-Fail)

The hard gate catches real overcommit risk. It sums only VM dedicated values (real reservations) plus ZFS ARC max plus a fixed host reserve:

# Hard gate: VM reservations + ARC + host reserve
ZFS_ARC_MAX_GB = 4       # /sys/module/zfs/parameters/zfs_arc_max
HOST_RESERVE_GB = 6       # kernel + QEMU overhead
HARD_GATE_CEILING_GB = 44 # safe ceiling for 62 GB physical

hard_gate_mb = vm_dedicated_total + (ZFS_ARC_MAX_GB * 1024) + (HOST_RESERVE_GB * 1024)

If the hard gate exceeds 44 GB, the CI build fails. The PR cannot be merged. This is intentional: a VM dedicated change that pushes past the ceiling is exactly the kind of change that caused the original freeze.

The ceiling (44 GB on a 62 GB host) leaves 18 GB of headroom for:

LXC actual usage (typically 8-12 GB across all containers)
Kernel and system overhead not captured in the reserve
Burst spikes from Ollama inference or Paperless OCR

Soft Check (Warn-Only)

The soft check sums LXC dedicated values and compares against physical RAM. If it exceeds 62 GB, a warning is printed — but the build does not fail. LXC ceilings are soft; summing them overstates real pressure.

# Soft check: LXC CT limits vs physical RAM (visibility only)
if lxc_dedicated_total > PHYSICAL_RAM_GB * 1024:
    print(f"WARN: LXC CT limits sum to {lxc_dedicated_total / 1024:.1f} GB, "
          f"over physical RAM ({PHYSICAL_RAM_GB} GB). "
          f"Check actual usage: pct exec <id> -- cat /sys/fs/cgroup/memory.current")

The soft check exists for visibility. If someone adds a new LXC with 32 GB dedicated and the sum crosses 100 GB, the warning fires — but it doesn't block the merge, because the actual usage is probably a fraction of the configured limit.

Why `floating` Is Ignored

VMs can have both dedicated (ceiling) and floating (balloon-driven floor). Under host memory pressure, Proxmox can deflate the balloon down to the floating minimum, freeing memory for other workloads.

The guard conservatively uses dedicated, not floating, because:

The balloon only deflates after pressure is detected — it doesn't prevent the pressure
A sudden allocation spike (Ollama loading a 26B model) can't wait for the balloon to deflate
The guard exists to catch pending risk, not to model steady-state usage

floating values are parsed and printed for visibility, never counted in the hard gate.

The CI Integration

The script runs in pre-commit hooks and GitHub Actions CI:

# .github/workflows/ci.yml
- name: Memory overcommit guard
  run: python scripts/check-host-memory-overcommit.py

A Terraform change that adds a new VM or increases a VM's dedicated value is checked against the ceiling before merge. If it pushes past 44 GB, the PR is blocked with a clear error message explaining why.

$ python scripts/check-host-memory-overcommit.py
REL-035 memory overcommit guard (two-tier model)

-- Hard gate (CI-fail): VM reservations + ARC + host reserve --
  VM `dedicated` sum (3 VMs): 36864 MB (36.0 GB)
  VM `floating` sum  (3 VMs): 40960 MB (40.0 GB) -- informational only, NOT counted
  + ZFS ARC max: 4 GB
  + host/hypervisor reserve: 6 GB
  = hard gate total: 47104 MB (46.0 GB)
  Ceiling: 44 GB

FAIL: hard gate total (46.0 GB) exceeds the 44 GB ceiling by 2.0 GB.

What It Prevents

The guard exists because mini — the single Proxmox host running this entire homelab — has no failover. A host freeze means every service is down simultaneously: k3s, databases, DNS, monitoring, backups. The only recovery is a hard power cycle.

REL-016 (the ZFS freeze) happened because RAM pressure pushed ZFS into a stall-wait state on the shared NVMe. The hard gate catches the most common path to that state: a VM dedicated change that leaves insufficient headroom for the kernel, ZFS, and LXC workloads.

It doesn't prevent every possible freeze — a runaway process inside a VM can still consume all its allocated memory and cause host pressure. But it prevents the planned overcommit: the Terraform change that accidentally pushes past the ceiling because someone added a new VM without checking the math. CPU has the same two-tier problem; see how cpu.units scheduling priority solved it for etcd.

Memory overcommit modeling is the same problem in Azure: Reserved VM instances guarantee physical memory allocation, while Burstable VMs share host memory and can be throttled. Mixing both in the same Availability Set without understanding the difference produces the same false-sense-of-security that my original guard had. The fix is the same: separate hard reservations from soft ceilings, and never sum them together.

GitOps for Firewall Rules: MikroTik + Terraform + Atlantis

david — Mon, 31 Aug 2026 13:31:54 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

My MikroTik RB5009 has no manual RouterOS configuration. Every firewall rule, every VLAN interface, every DHCP static lease, every NAT port-forward — all of it lives in Terraform, reviewed via Atlantis pull requests, and applied through the same GitOps workflow that manages the Kubernetes cluster. This is the delivery pipeline underneath the zero-trust default-deny firewall policy and the VLAN matrix — this article covers the Atlantis plumbing, not the rules themselves.

This article covers the full implementation: the locals block that drives the VLAN matrix, the place_before pattern for deterministic firewall ordering, and how a router's LED schedule ended up in Terraform.

View the complete homelab infrastructure source on GitHub 🐙

The Provider Setup

The routeros provider (v1.80+) connects to the MikroTik via the REST API:

# terraform/stacks/network/providers.tf
terraform {
  required_version = ">= 1.5"
  required_providers {
    routeros = {
      source  = "terraform-routeros/routeros"
      version = "~> 1.80"
    }
  }
  backend "s3" {
    bucket = "terraform-state"
    key    = "network/terraform.tfstate"
    endpoints = {
      s3 = "https://s3.woitzik.dev"
    }
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    skip_requesting_account_id  = true
  }
}

The S3 backend is Garage, the self-hosted S3-compatible storage running inside k3s. The Terraform state for the network stack lives on the same cluster that depends on the network — another circular dependency that's acceptable because a full cluster failure means the network is the least of your problems.

The VLAN Matrix

The locals block is the single source of truth for the entire network:

# terraform/stacks/network/main.tf
locals {
  homelab_vlans = {
    "vlan20-srv"    = 20
    "vlan30-dmz"    = 30
    "vlan40-iot"    = 40
    "vlan100-admin" = 100
  }

  rpi_port_mapping = {
    "ether6" = 20  # RPi 4B #1 (Keepalived Node A)
    "ether7" = 20  # RPi 4B #2 (Keepalived Node B)
  }

  proxmox_port = "ether5"

  # Port mapping for all VLAN-tagged trunks
  port_mapping = {
    "ether1" = 1   # WAN (FritzBox)
    "ether2" = 20  # k3s-11
    "ether3" = 20  # k3s-12
    "ether4" = 20  # k3s-13
    "ether5" = 20  # Proxmox host
    "ether6" = 20  # RPi #1
    "ether7" = 20  # RPi #2
    "sfp-sfpplus1" = 30  # DMZ switch
  }
}

Adding a new VLAN means one entry in homelab_vlans. Terraform auto-generates:

Bridge VLAN entries
DHCP server per VLAN
Firewall rules per VLAN
VLAN interface on the bridge

Moving a device between VLANs means changing one number in port_mapping. The firewall rules, DHCP leases, and bridge entries update automatically.

Deterministic Firewall Ordering

MikroTik evaluates firewall rules in order and stops at the first match. Terraform manages rule ordering via place_before references:

# terraform/stacks/network/firewall_deterministic.tf

# Final drop rule — must be last in the forward chain
resource "routeros_ip_firewall_filter" "fwd_99_drop_all" {
  action       = "drop"
  chain        = "forward"
  place_before = null  # explicit: this is the last rule
  comment      = "99: Global - Final Drop (Zero Trust Policy)"
}

# Anti-spoofing — must come before any accept rules
resource "routeros_ip_firewall_filter" "fwd_00b_anti_spoof" {
  action       = "drop"
  chain        = "forward"
  src_address  = "10.0.0.0/8"
  in_interface_list = "WAN"
  place_before = routeros_ip_firewall_filter.fwd_01_established.id
  comment      = "00b: Anti-Spoof - Drop internal src from WAN"
}

# Established/related — allows return traffic
resource "routeros_ip_firewall_filter" "fwd_01_established" {
  action       = "accept"
  chain        = "forward"
  connection_state = ["established", "related"]
  place_before = routeros_ip_firewall_filter.fwd_99_drop_all.id
  comment      = "01: Allow Established/Related"
}

The place_before attribute creates explicit ordering dependencies. Rule 00b must come before rule 01, which must come before rule 99. Terraform resolves these dependencies during plan, so the apply order matches the intended evaluation order.

Without place_before, Terraform would apply rules in dependency-graph order, which is not guaranteed to match the intended firewall evaluation order. A rule that should be evaluated first might be created last, allowing a broader rule to match traffic before the narrower, more specific rule is evaluated.

The full ruleset in firewall_deterministic.tf has 22 rules with explicit ordering. The firewall_extra.tf file has 19 additional rules for VPN access tiers, monitoring, and application-specific port forwards — all with place_before references to maintain correct evaluation order.

The Atlantis Flow

Every change to the network stack goes through Atlantis:

Edit terraform/stacks/network/*.tf
Push to a feature branch
Open a PR against main
Atlantis comments with the terraform plan output
Review the plan
Comment atlantis apply
Atlantis applies the changes to the MikroTik

The atlantis.yaml at the repo root defines the project:

version: 3
projects:
  - name: network
    dir: terraform/stacks/network
    workspace: default
    autoplan:
      when_modified: ["*.tf", "*.tfvars"]
      enabled: true

The plan output shows exactly what will change:

# routeros_ip_firewall_filter.fwd_04a_monitoring will be updated in-place
~ resource "routeros_ip_firewall_filter" "fwd_04a_monitoring" {
      action      = "accept"
      chain       = "forward"
  ~   dst_port    = "9100" -> "9100,9090"
      src_address = "10.0.20.0/24"
      comment     = "04a: SRV - Prometheus scrape to MGMT"
  }

No surprise changes. Every rule modification is visible in the PR before it touches the live router.

What Goes in Terraform vs. What Doesn't

In Terraform:

All firewall filter rules (input + forward chains)
VLAN interfaces and bridge matrix
DHCP static leases
NAT port-forwards
QoS traffic shaping
SNMP community configuration
Power LED scheduling (yes, really)

Not in Terraform:

RouterOS user management (done once, rarely changes)
Certificate renewals (handled by the router's own scheduler)
Custom scripts for monitoring (run via SNMP, not Terraform)

The principle: if it affects traffic flow or network behavior, it's in Terraform. If it's one-time configuration or the router's own maintenance, it stays in RouterOS.

The LED Night-Mode

The MikroTik RB5009 has a power LED that's bright enough to be annoying in a dark room. RouterOS supports LED scheduling via the system scheduler. This ended up in Terraform:

resource "routeros_system_scheduler" "led_night_mode_off" {
  name     = "led-off"
  start_time = "23:00:00"
  interval  = "00:24:00"
  on_event = "/system led set led1 state=off"
}

resource "routeros_system_scheduler" "led_night_mode_on" {
  name     = "led-on"
  start_time = "07:00:00"
  interval  = "00:24:00"
  on_event = "/system led set led1 state=on"
}

Two scheduler entries. LED off at 23:00, on at 07:00. Managed via Terraform, reviewed via PR, applied via Atlantis. It's the most trivial thing in the entire network stack, and it's also the change that made my partner the happiest.

The Anti-Patterns

Two things I deliberately avoid:

Never create resources directly via RouterOS API. Every resource must be created through Terraform. If I need to debug a live issue and make a direct API change, I immediately add a moved {} block or import {} block to bring that resource under Terraform management. This prevents the "77 rules, Terraform knows about 22" problem I wrote about earlier.
Never terraform apply locally. All applies go through Atlantis PRs. This creates an audit trail, forces me to review every change, and prevents accidental modifications during debugging sessions.

Firewall-as-code via Terraform is the same pattern in Azure: NSG rules, Azure Firewall Policy rule collections, and Application Security Groups are all Terraform resources. The difference is that Azure's NSG API handles rule ordering automatically based on priority numbers, while MikroTik requires explicit place_before dependencies. Both approaches achieve the same goal: firewall rules that are reviewed, version-controlled, and auditable.