DEV Community: david

Vault Auto-Unseal Without Cloud KMS: The Polling Sidecar Pattern

david — Mon, 24 Aug 2026 07:34:21 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

HashiCorp Vault Enterprise has auto-unseal: a sealed Vault automatically unseals using a cloud KMS (AWS KMS, Azure Key Vault, GCP Cloud KMS). Vault OSS doesn't. Every time a Vault pod restarts — node failure, OOMKill, Kubernetes rescheduling — someone has to manually unseal it with vault operator unseal using the unseal keys.

On a homelab cluster where Vault backs ExternalSecrets for 25+ services, a sealed Vault means every ExternalSecret refresh fails. Authelia can't start (no hmac-secret), Open WebUI can't start (no WEBUI_SECRET_KEY), and half the cluster sits in init-container loops waiting for secrets that Vault can't provide.

The fix: a polling sidecar that auto-unseals Vault OSS without a KMS. The trade-off is a collapsed security boundary — but for a homelab, it's the right trade-off.

View the complete homelab infrastructure source on GitHub 🐙

The Sidecar

The vault-unseal Deployment runs as a separate pod in the vault namespace, polling every 5 seconds:

# kubernetes/system/vault/unseal.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault-unseal
  namespace: vault
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vault-unseal
  template:
    spec:
      serviceAccountName: vault-unseal
      containers:
        - name: vault-unseal
          image: hashicorp/vault:1.21.4
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                # Check if Vault is sealed
                SEALED=$(vault status -format=json | jq -r .sealed)
                if [ "$SEALED" = "true" ]; then
                  # Read unseal keys from the Kubernetes Secret
                  for i in 1 2 3; do
                    KEY=$(kubectl get secret vault-unseal-keys \
                      -n vault -o jsonpath="{.data.key$i}" | base64 -d)
                    vault operator unseal "$KEY"
                  done
                  echo "Vault unsealed at $(date)"
                fi
                sleep 5
              done
          env:
            - name: VAULT_ADDR
              value: "http://vault.vault.svc.cluster.local:8200"
          resources:
            requests:
              cpu: 10m
              memory: 32Mi
            limits:
              cpu: 50m
              memory: 64Mi

The pod needs kubectl access to read the unseal keys from a Kubernetes Secret. The vault-unseal ServiceAccount has a Role that grants get on the vault-unseal-keys Secret only.

The Security Trade-off

In a KMS-based auto-unseal setup, the unseal keys are never stored anywhere accessible to the cluster. The KMS holds the master key, and Vault uses it to decrypt the master key that seals the storage. An attacker who compromises the cluster can't unseal Vault because the KMS key is outside the cluster boundary.

In the polling sidecar pattern, the unseal keys are stored in a Kubernetes Secret (vault-unseal-keys). Anyone who can read that Secret — through kubectl, through a compromised pod with the right ServiceAccount, through an etcd backup — can unseal Vault.

This collapses the security boundary: Vault's unseal protection becomes "Kubernetes RBAC on one Secret" instead of "cloud KMS with its own IAM policy." For a homelab, this is acceptable because:

The cluster is not exposed to untrusted users
The threat model is "protect against accidental unseal, not nation-state attacker"
Manual unseal after every restart is operationally unsustainable

For production, use KMS-based auto-unseal. For a homelab where the alternative is "Vault stays sealed until I notice," the sidecar is the pragmatic choice.

Why 5 Seconds

The initial implementation polled every 30 seconds. This created a window where Vault was sealed but the sidecar hadn't tried to unseal it yet. During that window:

ExternalSecret refresh requests failed
Authelia's init container couldn't read hmac-secret
Any service that depends on Vault-backed secrets was stuck

30 seconds of cluster-wide secret unavailability on every Vault restart. Tightening to 5 seconds reduced the seal window to an acceptable range — most Vault restarts complete unseal within 5 seconds, and the downstream impact is minimal.

The cost: the sidecar makes one vault status call and potentially three vault operator unseal calls every 5 seconds. On Vault's API, this is negligible — it's health-check-level traffic.

The ExternalSecret Dependency Chain

The real reason Vault auto-unseal matters: every ExternalSecret in the cluster depends on Vault being unsealed.

Vault sealed
  → ExternalSecret refresh fails
    → Kubernetes Secrets not updated
      → Pods using those secrets start with stale/missing data
        → Authelia can't start (no hmac-secret)
        → Open WebUI can't start (no WEBUI_SECRET_KEY)
        → Paperless can't start (no database password)

The chain reaction is invisible until you check pod logs and see secret "authelia-secrets" not found or connection refused to Postgres (because the password never synced from Vault).

Before the sidecar, I'd come back to a sealed Vault after a node restart and spend 15 minutes manually unsealing with three key shares while half the cluster sat in CrashLoopBackOff. The sidecar turned a 15-minute manual operation into a 5-second automatic one.

The Network Policy

Vault's ingress is locked down:

# kubernetes/system/vault/network-policies.yml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vault-allow-intra-namespace
  namespace: vault
spec:
  podSelector: {}
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - namespaceSelector: {}
          podSelector: {}
    - from:
        - namespaceSelector:
            matchLabels: {}
          podSelector:
            matchLabels:
              app.kubernetes.io/name: external-secrets
      ports:
        - port: 8200

Default-deny ingress on the vault namespace. Only two sources can reach Vault's API port 8200: other pods in the vault namespace (the unseal sidecar), and External Secrets Operator pods. Everything else is blocked.

This is the minimum viable network segmentation for Vault: it needs to be reachable by ESO for secret syncing and by the unseal sidecar for auto-unseal, but nothing else needs direct Vault API access.

What I'd Change

Use Vault's built-in auto-unseal with a cloud KMS if available. The sidecar is a workaround for Vault OSS limitations. If you're running Vault Enterprise or can tolerate the cost of a cloud KMS, use it. The sidecar exists because my homelab doesn't have a KMS. Vault's configuration (policies, auth roles) is a separate concern from unsealing — see the staged Terraform migration for how that part is managed without touching unseal keys.
Store unseal keys in a more secure backend. The Kubernetes Secret is the weakest link. An improvement would be to store the keys in an HSM or a separate, more restricted secret backend. But at that point, you've basically built KMS-based auto-unseal from scratch.
Add monitoring on the sidecar. An alert when Vault transitions from sealed to unseal would provide visibility into restart frequency and sidecar health.

Vault auto-unseal without KMS is the same problem as managing encryption keys in environments without HSMs: you're trading security boundary strength for operational practicality. Azure Key Vault Managed HSM provides FIPS 140-2 Level 3 key protection — but it costs money and adds complexity. For non-production environments, the polling sidecar gives you 90% of the operational benefit at 10% of the security cost. The key is knowing which trade-off you're making.

Chaos Mesh in a Homelab: Weekly Pod-Kills on a Single-Host Cluster

david — Mon, 24 Aug 2026 07:13:35 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Chaos engineering sounds like a production-only practice — inject failures, measure impact, improve resilience. But the biggest value of chaos testing isn't catching production failures. It's finding the problems you didn't know existed, in infrastructure you thought was solid, before they find you.

I run two weekly chaos experiments on my single-host k3s cluster: a pod-kill every Sunday at 03:00 UTC, and a 100ms network latency injection at 03:30 UTC. Both are scoped to pods labeled chaos-kill: enabled in the apps namespace. Here's what they've taught me.

View the complete homelab infrastructure source on GitHub 🐙

The Setup

Chaos Mesh runs in the chaos-mesh namespace, deployed via Helm chart v2.8.3:

# kubernetes/system/chaos-mesh/application.yml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: chaos-mesh
  namespace: argocd
spec:
  source:
    repoURL: https://charts.chaos-mesh.org
    chart: chaos-mesh
    targetRevision: 2.8.3
    helm:
      values:
        chaosDaemon:
          runtime: containerd
          socketPath: /run/k3s/containerd/containerd.sock

The socketPath is critical: k3s uses its own containerd socket, not the standard Docker or containerd paths. If you point Chaos Mesh at the wrong socket, pod-kill experiments silently fail — the controller reports success but no pods are actually killed.

The two Schedule resources:

# Weekly pod-kill — Sunday 03:00 UTC
apiVersion: chaos-mesh.org/v1alpha1
kind: Schedule
metadata:
  name: weekly-pod-kill
  namespace: chaos-mesh
spec:
  schedule: "0 3 * * 0"
  type: PodChaos
  podChaos:
    action: pod-kill
    mode: one
    selector:
      labelSelectors:
        chaos-kill: "enabled"
      namespaces: ["apps"]
    gracePeriod: 0
---
# Weekly network latency — Sunday 03:30 UTC
apiVersion: chaos-mesh.org/v1alpha1
kind: Schedule
metadata:
  name: weekly-network-latency
  namespace: chaos-mesh
spec:
  schedule: "30 3 * * 0"
  type: NetworkChaos
  networkChaos:
    action: delay
    mode: all
    selector:
      labelSelectors:
        chaos-network: "enabled"
      namespaces: ["apps"]
    delay:
      latency: "100ms"
      jitter: "10ms"
    duration: "5m"
    direction: to

The mode: one on the pod-kill means exactly one pod matching the label is killed per experiment. mode: all on the network latency applies to all pods with the label. The gracePeriod: 0 on pod-kill means immediate termination — no graceful shutdown, which is the realistic failure mode (a kernel panic, a power loss, a node crash).

What Broke: First Week

The first pod-kill Sunday killed one Authelia pod. Expected behavior: Kubernetes reschedules it within seconds. What actually happened:

Pod killed at 03:00:00 UTC
Kubernetes schedules replacement at 03:00:02 UTC
Replacement starts wait-for-vault-secret init container
Init container polls Vault for ExternalSecret sync at 30s intervals
Authelia fully ready at 03:01:30 UTC — 90 seconds of downtime

Ninety seconds for Authelia to recover from a pod-kill. The init container is the bottleneck: it waits for Vault to provide the hmac-secret, OIDC keys, and session secrets before the main container starts. During those 90 seconds, any service that checks Authelia for authentication returns 501 — the Traefik middleware can't reach Authelia's /api/verify endpoint.

The fix wasn't to make Authelia faster. The fix was to recognize that 90 seconds of Authelia downtime is acceptable for a single-replica-killed scenario, but unacceptable for a two-replica scenario where both pods are killed simultaneously. The PDB (minAvailable: 1) prevents simultaneous kills — Chaos Mesh respects PodDisruptionBudgets. Without the PDB, both pods could be killed in the same experiment window.

What Broke: Network Latency

The 100ms latency injection was more insidious. It didn't break anything immediately. Instead, it exposed timing-dependent behavior that was invisible under normal network conditions:

Velero backup duration increased by 40%. Velero's Kopia sidecar communicates with the Garage S3 endpoint over the cluster network. Adding 100ms per request multiplied across thousands of file operations extended the backup window from ~12 minutes to ~20 minutes.

ArgoCD sync operations became sluggish. ArgoCD's repo-server fetches manifests from the git repo, applies diffs, and syncs. Each step involves network calls that now had 100ms added. Syncs that normally took 5 seconds took 15-20 seconds.

Uptime Kuma monitors flickered. Uptime Kuma's HTTP monitors expect sub-second response times. The 100ms added by Chaos Mesh pushed some monitors past their threshold, generating false-positive "service down" alerts.

None of these are failures. They're performance degradation under adverse conditions. But they reveal the hidden assumption in every service's timeout and retry configuration: "the network is fast." When it isn't — because of a real network issue, a noisy neighbor, a congested switch — services that work fine under normal conditions start failing in unexpected ways.

What It Taught

1. PDBs aren't optional

Before Chaos Mesh, the PodDisruptionBudgets for Authelia and cloudflared were theoretical — "we have them because best practices say we should." After the first pod-kill confirmed that the PDB actually prevented simultaneous kills, they became load-bearing infrastructure.

2. Init containers are single points of failure

Every ExternalSecret-backed deployment has an init container that waits for Vault. If Vault is slow, sealed, or unreachable, the init container blocks the entire pod startup. The 90-second Authelia recovery time is entirely dominated by this init container. A faster health check or a cached secret fallback would reduce recovery time.

3. Latency exposes timeout assumptions

Every service has implicit assumptions about network latency. When those assumptions are violated, services don't crash — they degrade. The degradation is harder to debug than a crash because everything looks healthy in the logs. The only signal is slower response times and increased error rates that don't quite reach alerting thresholds.

4. Chaos testing on a homelab isn't overkill

The cluster has no SLA. Nobody is paying for uptime. But the same workloads (Postgres, Vault, Authelia) run in production environments everywhere. Finding that a pod-kill takes 90 seconds to recover — on a homelab where the consequence is "I can't log in for a minute and a half" — is infinitely better than finding it in production where the consequence is a customer-facing outage. It's the same reasoning behind treating a cascading OOM-kill/DNS-storm failure as worth a full root-cause writeup even though nobody was paged.

Chaos engineering at enterprise scale uses the same tools: Azure Chaos Studio for VM and AKS fault injection, Azure Load Testing for performance baseline, and Azure Monitor for measuring blast radius. The principle is identical — inject realistic failures in a controlled environment, measure the impact, fix what breaks. The only difference is that Azure Chaos Studio charges per experiment, so you want your homelab practice run to be thorough.

Cloudflare Tunnel Without Opening a Single Firewall Port

david — Mon, 17 Aug 2026 07:10:16 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Three services in my homelab need external access: Immich for family photo backup, Atlantis for Terraform webhooks, and Jellyfin for media. None of them have a single inbound firewall rule — the MikroTik firewall stays default-deny with no WAN-facing accept rules at all. Cloudflare Tunnel handles all external traffic through an outbound-only connection, and split-DNS on AdGuard ensures LAN clients reach services directly without leaving the network.

View the complete homelab infrastructure source on GitHub 🐙

The Architecture

Cloudflare Tunnel works by running a cloudflared daemon inside the cluster that maintains an outbound-only connection to Cloudflare's edge. External requests hit Cloudflare, get routed through the tunnel to the origin service, and back. No inbound ports, no port forwarding, no attack surface on the WAN side.

The Terraform configuration defines the tunnel ingress rules:

# terraform/stacks/cloudflare/main.tf
resource "cloudflare_zero_trust_tunnel_cloudflared_config" "homelab" {
  tunnel_id = var.tunnel_id
  account_id = var.account_id
  config {
    ingress {
      hostname = "atlantis.woitzik.dev"
      service  = "https://traefik.apps.svc.cluster.local:443"
      origin_request {
        origin_server_name = "atlantis.woitzik.dev"
        no_tls_verify     = false
      }
    }
    ingress {
      hostname = "photos.woitzik.dev"
      service  = "http://immich-server.apps.svc.cluster.local:2283"
      origin_request {
        no_tls_verify    = false
        chunked_encoding = true
      }
    }
    ingress {
      hostname = "media.woitzik.dev"
      service  = "http://ct-srv-jellyfin-01.dmz.woitzik.dev:8096"
      origin_request {
        no_tls_verify = false
      }
    }
    ingress {
      service = "http_status:404"
    }
  }
}

The catch-all http_status:404 is mandatory — Cloudflare Tunnel requires a final rule that catches unmatched hostnames.

Split-DNS on AdGuard

The tunnel creates a split-brain DNS problem. External requests go through Cloudflare's anycast IPs. Internal requests (LAN) should go directly to the Traefik VIP, bypassing Cloudflare entirely. This means photos.woitzik.dev needs to resolve to different IPs depending on where the client is.

AdGuard Home handles this with wildcard overrides:

# AdGuard rewrites
# Default: *.woitzik.dev → 10.0.20.200 (Traefik VIP)
# Override: photos.woitzik.dev → Cloudflare anycast (172.67.137.91, 104.21.38.184)

The *.woitzik.dev → 10.0.20.200 wildcard covers all internal services. The specific override for photos.woitzik.dev points to Cloudflare's anycast IPs, so mobile clients on cellular networks get routed through the tunnel while LAN clients go direct.

The gotcha: AdGuard's wildcard applies to the search domain too. If the Proxmox host's search domain is woitzik.dev, every internal hostname query goes through the wildcard — including pve.woitzik.dev, pbs.woitzik.dev, and other services that should never leave the LAN. This was caught during initial setup: PTR queries for k3s pod IPs were being forwarded to the FritzBox because AdGuard's search domain configuration leaked DNS traffic outward.

The Chunked Encoding Fix

Large photo and video uploads to Immich through Cloudflare Tunnel were failing with ECONNRESET. The uploads would complete 60-80% then drop the connection.

The root cause: Cloudflare's proxy buffers the entire request body before forwarding to the origin. For multi-GB video uploads, this exceeds Cloudflare's buffer limit and the connection resets.

The fix in the Terraform tunnel config:

origin_request {
  chunked_encoding     = true
  write_timeout        = 600  # seconds
  read_timeout         = 120  # seconds
}

chunked_encoding = true tells cloudflared to stream the request body to the origin in chunks instead of buffering the entire payload. write_timeout = 600s gives large uploads up to 10 minutes to complete.

This was the Immich-specific fix documented in CHANGELOG v0.8.0. The same issue can affect any service behind Cloudflare Tunnel that accepts large file uploads — Nextcloud, Paperless, or any service accepting multipart form data above ~100MB.

Atlantis Through Traefik

Atlantis needs to receive GitHub webhooks for Terraform PR events. The naive approach: expose Atlantis directly via Cloudflare Tunnel. The problem: Atlantis has no built-in authentication — anyone who knows the URL can trigger plans and applies.

The fix: route Atlantis through Traefik first, which applies the Authelia ForwardAuth middleware before the request reaches Atlantis:

ingress {
  hostname = "atlantis.woitzik.dev"
  service  = "https://traefik.apps.svc.cluster.local:443"
  origin_request {
    origin_server_name = "atlantis.woitzik.dev"
  }
}

The Cloudflare Tunnel points at Traefik, not Atlantis. Traefik's IngressRoute for Atlantis includes the Authelia middleware:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: atlantis
  namespace: apps
spec:
  entryPoints: [websecure]
  routes:
    - match: Host(`atlantis.woitzik.dev`)
      kind: Rule
      middlewares: [{name: authelia}]
      services:
        - name: atlantis
          port: 4141
  tls:
    secretName: wildcard-woitzik-dev-tls

GitHub webhooks bypass Authelia because the webhook source IPs are known and can be allowlisted separately. But the Atlantis web UI — the plan output, the apply buttons — requires Authelia authentication.

This pattern (Cloudflare Tunnel → Traefik → Authelia → service) is the same for every external service. The tunnel handles ingress, Traefik handles routing and TLS, Authelia handles authentication. Each layer does one thing.

DNS Record Management

Cloudflare DNS records for the tunnel CNAMEs are Terraform-managed:

resource "cloudflare_dns_record" "tunnel_photos" {
  zone_id = var.zone_id
  name    = "photos"
  type    = "CNAME"
  content = "abc123.cfargotunnel.com"
  proxied = true
  ttl     = 1
}

The proxied = true means traffic goes through Cloudflare's proxy, which provides DDoS protection, rate limiting, and WAF rules. ttl = 1 is automatic for proxied records.

The auth DNS record (CNAME to home.woitzik.dev) is proxied = false — it's an internal-only record that doesn't need Cloudflare's proxy layer. Same for the Minecraft playit.gg record.

Cloudflare Tunnel's zero-inbound-port model is the same pattern as Azure Private Link and Azure Front Door: external traffic enters through a managed proxy, never touches your firewall directly, and the origin service only needs outbound connectivity. The split-DNS complexity is the same as Azure DNS Private Zones — internal resolution goes through private endpoints, external resolution through public DNS. Different tools, identical architecture.

K3s on Raspberry Pi: Why I Said No (And You Probably Should Too)

david — Sat, 15 Aug 2026 10:03:33 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

The idea was elegant: three Raspberry Pi 5 nodes running k3s control-plane + embedded etcd, replacing the three Proxmox VMs. Lower power consumption, physical separation from the main host, and a genuine HA cluster spread across real hardware.

It didn't work. The failure wasn't dramatic — no kernel panic, no cluster death. It was a slow accumulation of fragility that made the cluster less reliable than the single-node setup it replaced.

View the complete homelab infrastructure source on GitHub 🐙

The Attempt

The plan (ADR-014, Option B): run k3s embedded etcd on three Raspberry Pi 5 (8GB)* nodes. The Pis already handle DNS (AdGuard + Unbound) and Keepalived VIP. Adding k3s control-plane seemed like a natural extension.

The k3s cluster spec:

3x Raspberry Pi 5 (8GB)
256GB microSD cards (A2 rated)
Gigabit Ethernet via USB 3.0 adapter (Pi 5's native Ethernet is limited)
k3s v1.31 with embedded etcd

Why It Failed: SD Card Fragility

etcd is write-heavy. Every Kubernetes API operation — pod scheduling, configmap updates, secret rotations — generates etcd writes. The k3s embedded etcd writes continuously to the local filesystem.

SD cards have limited write endurance. The A2-rated cards I used are rated for ~150 MB/s sequential write, but the random 4K write IOPS that etcd generates are a different story. Under sustained write load, the SD card's write amplification factor increases, the garbage collection cycle can't keep up, and write latency spikes.

The symptom: etcd response times would occasionally jump from 10ms to 500ms+ for no apparent reason. No CPU load, no network congestion, no memory pressure. Just slow disk writes. On a NVMe-backed VM, etcd writes complete in microseconds. On an SD card, they're orders of magnitude slower.

etcd has a built-in leader election timeout (default 5s). When etcd write latency exceeds the election timeout, the leader steps down and a new election starts. If the new leader is also on an SD card with the same write latency problem, the election can fail too. The result: brief periods where the Kubernetes API server is unavailable, even though all nodes are "healthy."

Why It Failed: Network Latency

The three Pis communicate over Gigabit Ethernet via USB 3.0 adapters. The adapter adds ~0.5ms of latency per hop compared to native Gigabit. For etcd consensus, which requires a majority of nodes to acknowledge each write, the added latency compounds:

Write to leader: 0.5ms
Leader replicates to 2 followers: 0.5ms × 2 = 1ms
Followers acknowledge: 0.5ms × 2 = 1ms
Total round-trip: ~2.5ms

Compare to three VMs on the same NVMe-backed host: ~0.1ms round-trip. The 25x latency increase doesn't matter for normal API operations, but it matters during high-write periods or leader elections, exactly when low latency is most critical.

Why It Failed: Resource Contention

The Pis were already running AdGuard + Unbound for network DNS, Keepalived for the VIP, and node_exporter for monitoring. Adding k3s control-plane + etcd meant four services competing for the same CPU, memory, and — critically — the same SD card.

When AdGuard's DNS cache expired and refreshed simultaneously with an etcd compaction cycle, both hit the SD card at once. The resulting I/O contention produced the same pattern as the Proxmox host freeze: etcd write latency spikes → leader election → temporary API unavailability.

The Deliberate Reversion

After three weeks of monitoring, the k3s-on-Pi cluster was less reliable than the single-node setup it replaced. The decision (ADR-014): revert to a single k3s control-plane on the Proxmox VM (vm-srv-k3s-11), with two agent-only worker nodes.

# vm-srv-k3s-11: control-plane + etcd (sole server)
# vm-srv-k3s-12: agent only
# vm-srv-k3s-13: agent only

The single-server setup has a known limitation: if k3s-11 goes down, the entire cluster is down. No failover, no HA. But "down" in a homelab context means "inaccessible for a few minutes while I restart the VM" — not "data center outage affecting thousands of users."

The trade-off: reliability (single NVMe-backed VM) over availability (three SD-card-backed Pis). For a homelab, reliability is the right priority.

What the Pis Do Instead

The Raspberry Pis remain in the rack, running DNS and Keepalived — exactly what they're good at:

AdGuard Home: DNS filtering for the entire network, low write volume, well within SD card endurance
Unbound: Recursive DNS resolver, almost entirely read operations after cache warm-up
Keepalived: Active/passive VIP failover, heartbeat-only, negligible I/O
node_exporter: System metrics, read-only

These workloads are low-write, low-latency-sensitive, and can tolerate brief interruptions without affecting the cluster. The Keepalived VIP failing over to the other Pi is a 1-second blip. The k3s API server going down for 30 seconds during an etcd election breaks every kubectl command and ArgoCD sync.

The General Lesson

etcd's requirements are specific: low-latency, high-endurance storage with consistent write performance. NVMe provides this. SATA SSDs provide this (mostly). SD cards and USB-attached storage do not, because their write latency is unpredictable under sustained load.

If you're running k3s on Raspberry Pis, the stable configuration is: Pis as agent-only workers, control-plane on a VM or dedicated x86 box with proper storage. The Pis handle DNS, monitoring, and lightweight workloads — things where an SD card's write endurance is adequate and a brief interruption doesn't cascade into a cluster-wide event.

The same logic applies to etcd on cloud VMs: don't put etcd on Standard HDD. The IOPS and latency guarantees of Premium SSD or Ultra Disk exist specifically because etcd's consensus protocol requires consistent, low-latency writes. The Pi's SD card is the cloud equivalent of a Standard HDD — it works until you need consistent performance under load.

K3s on ARM is production-ready for worker nodes. The control-plane decision is about storage performance and write consistency, not CPU architecture. Azure's AKS control plane runs on the same principle: the control-plane nodes use Premium SSD managed disks precisely because etcd needs consistent, low-latency writes. The underlying hardware doesn't matter — the I/O guarantees do.

Azure Logic App Standard: Private Storage Needs Four Private Endpoints

david — Sat, 15 Aug 2026 09:51:09 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Logic App Standard is the go-to hosting model when your workflows need VNet integration and a predictable App Service Plan. In a hardened Azure landing zone, that means one thing: a Storage Account that is not reachable from the public internet, fronted by Private Endpoints.

The deployment succeeded. The Resource Group, the Storage Account, the private endpoints, the DNS zones - everything planned cleanly and applied without a single error. Then the workflow host refused to start. Every single boot ended in the same 403 Forbidden, roughly 600 milliseconds after the host registered its blob webhook endpoint.

This is the postmortem: why it happened, why four plausible fixes did not help, and how the generic rebuild now bakes the correct configuration in by default.

View the base source code on GitHub 🐙

The Symptom

The portal designer showed a generic failure - renderComponentIntoRoot, Workflow validation failed - none of which pointed anywhere useful. A direct API call against the host runtime revealed the real error:

Encountered an error (ServiceUnavailable) from host runtime.

The Kudu host status endpoint (/hostruntime/admin/host/status) confirmed the host was not just having a bad moment:

{
  "state": "Error",
  "errors": [
    "Microsoft.WindowsAzure.ResourceStack: Unexpected HTTP status code 'Forbidden'. The remote server returned an error: (403) Forbidden."
  ]
}

The Forbidden came from Storage itself, and it came back fast. That timing was the first real clue: no timeout, no firewall blackhole, no DNS hang - a prompt, deliberate rejection.

The Root Cause

A Logic App Standard is a Functions host under the hood - a functions app hosting workflow apps. Its internal host runtime does not just need Storage for your workflow content. It consumes four Storage subresources during normal operation:

Subresource	Used for
File	Content share - `host.json`, `connections.json`, workflow definitions
Blob	Extension-bundle cache, application logs
Queue	Internal WebJobs coordination - scale controller, trigger bookkeeping
Table	Internal WebJobs metadata

Our wrapper built private endpoints for Blob and File - the two that look obviously necessary, because the File share literally holds the content and Blob holds the bundle cache. Queue and Table were forgotten.

The consequence is subtle and nasty. DNS for *.queue.core.windows.net and *.table.core.windows.net resolved to the public IP - there is no private endpoint, so no private DNS record. But the Storage Account had publicNetworkAccess disabled. Every request the host made to Queue or Table over the public IP was rejected by Storage itself with 403 Forbidden - instantly. That is why the failure appeared exactly ~600 ms after boot: Storage denies fast, it does not hang.

Why Four Plausible Fixes Failed

Each attempt was reasonable, and each targeted the wrong layer:

RG Contributor for the Managed Identity. RBAC was never the problem. The Storage network firewall does not consult RBAC roles - it only evaluates network origin plus keys/auth. Adding permissions changed nothing.
vnet_route_all_enabled = false. This flag only controls whether outbound internet traffic is routed through the VNet. Queue/Table calls never went through the VNet in the first place - they were direct public calls caused by wrong DNS resolution. The flag was for a different layer of the problem.
Storage data-plane RBAC (Blob/Queue/Table Data Contributor). Permission is useless when the request is rejected at the network layer before RBAC is even evaluated.
Finally, Application Insights. The Kudu logs showed only the short error with no stack trace. Attaching App Insights temporarily revealed the full exception stack and the exact failing method (WorkflowExtensionProvider.Initialize). Only then was the real dependency visible.

The lesson from the four failed fixes: when a private Storage Account rejects a request with 403, the problem is almost never identity. It is reachability - specifically, which DNS name resolves to which IP.

The Fix

Two additional Private Endpoints on the same Storage Account, plus their Private DNS zone integration:

pe-st-queue → privatelink.queue.core.windows.net
pe-st-table → privatelink.table.core.windows.net

The private DNS zones already existed centrally; they just were not attached to this Storage Account. Once Queue and Table resolved to private IPs, the host booted cleanly on the next restart.

The memory anchor for next time: a Logic App Standard or Function App with private Storage needs all four Storage subresources behind Private Endpoints - not just the ones that look content-related. Queue and Table are pure host-internal infrastructure. They show up in no obvious configuration, which is exactly why they get forgotten.

The Generic Rebuild

Instead of leaving the fix in a wrapper module, the generic template now creates all four by default. The private endpoints are a single for_each over the subresource list:

locals {
  storage_subresources = ["blob", "file", "queue", "table"]
}

resource "azurerm_private_endpoint" "storage" {
  for_each            = toset(local.storage_subresources)
  name                = "pe-${local.storage_name}-${each.key}"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
  subnet_id           = azurerm_subnet.private_endpoints.id

  private_service_connection {
    name                           = "psc-${local.storage_name}-${each.key}"
    private_connection_resource_id = azurerm_storage_account.main.id
    subresource_names              = [each.key]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name                 = "dns-${each.key}"
    private_dns_zone_ids = [azurerm_private_dns_zone.main[each.key].id]
  }
}

Adding a sixth subresource (e.g. dfs for Data Lake) is now one list entry instead of a copy-pasted resource block.

The DNS zones are created and linked to the VNet in the same pass, so *.queue.core.windows.net resolves privately without touching your DNS infrastructure:

locals {
  private_dns_zones = {
    blob  = "privatelink.blob.core.windows.net"
    file  = "privatelink.file.core.windows.net"
    queue = "privatelink.queue.core.windows.net"
    table = "privatelink.table.core.windows.net"
    sites = "privatelink.azurewebsites.net"
  }
}

resource "azurerm_private_dns_zone" "main" {
  for_each            = local.private_dns_zones
  name                = each.value
  resource_group_name = azurerm_resource_group.main.name
}

resource "azurerm_private_dns_zone_virtual_network_link" "main" {
  for_each              = local.private_dns_zones
  name                  = "link-${azurerm_virtual_network.main.name}"
  resource_group_name   = azurerm_resource_group.main.name
  private_dns_zone_name = azurerm_private_dns_zone.main[each.key].name
  virtual_network_id    = azurerm_virtual_network.main.id
}

Two further details worth stealing from the fix:

1. Order the dependency. The Logic App must not boot before its private endpoints exist, otherwise the first scale-up races the network configuration:

resource "azurerm_logic_app_standard" "main" {
  # ...

  public_network_access     = "Disabled"
  virtual_network_subnet_id = azurerm_subnet.integration.id

  app_settings = merge(
    {
      "WEBSITE_CONTENTOVERVNET" = "1"
      "WEBSITE_VNET_ROUTE_ALL"  = "1"
    },
    var.app_settings
  )

  depends_on = [
    azurerm_private_endpoint.storage["blob"],
    azurerm_private_endpoint.storage["file"],
    azurerm_private_endpoint.storage["queue"],
    azurerm_private_endpoint.storage["table"],
  ]
}

WEBSITE_CONTENTOVERVNET = 1 tells the host to serve its own content share over the VNet instead of the public file endpoint.

2. Bootstrap the file share. A freshly deployed Standard Logic App boots against an empty share, and the workflow host and the portal designer both trip over that. The template uploads a minimal host.json / connections.json into site/wwwroot during the first apply - empty definition, but a valid host contract:

resource "local_file" "host_json" {
  filename = "${path.module}/.bootstrap-content/host.json"
  content = jsonencode({
    version = "2.0"
    extensionBundle = {
      id      = "Microsoft.Azure.Functions.ExtensionBundle.Workflows"
      version = "[1.*, 2.0.0)"
    }
  })
}

The Checklist

When you wire a Logic App Standard or Function App to a private Storage Account:

File private endpoint - content share
Blob private endpoint - bundle cache and logs
Queue private endpoint - host-internal coordination
Table private endpoint - host-internal metadata
Sites private endpoint on the Logic App itself (privatelink.azurewebsites.net)
Private DNS zones for all five, linked to the VNet
public_network_access = "Disabled" on the Storage Account and the Logic App
WEBSITE_CONTENTOVERVNET = 1 so content is served over the VNet
depends_on from the Logic App to the private endpoints

The Takeaway

A private endpoint per Storage subresource is not a nice-to-have for Logic Apps Standard - it is a hard requirement of the host runtime. The trap is that Queue and Table are invisible in every configuration file and every blog tutorial, so they get skipped until a 403 Forbidden starts appearing in Kudu logs at every boot.

The good news: the fix is deterministic, and it is now encoded in the template rather than remembered by an engineer who already paid the price once.

Velero Said Backups Succeeded. The Data Was Never There.

david — Mon, 10 Aug 2026 08:15:50 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Velero reported "Completed" on every daily backup. The schedule ran on time, the S3 uploads succeeded, the backup size looked reasonable. I never verified the contents — because "Completed" means completed, right?

On 2026-06-19, I needed to restore a PVC. The backup existed in Garage S3. The restore completed. The PVC was empty.

Postgres data, Vaultwarden passwords, Paperless documents, Nextcloud files — none of it was there. Velero had been faithfully backing up Kubernetes manifests for weeks while completely ignoring the actual data in PersistentVolumeClaims.

View the complete homelab infrastructure source on GitHub 🐙

What Velero Actually Backs Up

Velero has two backup modes for PersistentVolumes:

Snapshot-based (default): Takes a CSI snapshot of the PV. Works for cloud providers with snapshot APIs (EBS, Azure Disk). Doesn't work for NFS, local-path, or any storage class without a snapshot provider.
Filesystem-based (--default-volumes-to-fs-backup): Runs a Kopia or Restic sidecar that copies the actual files from the PVC. Works with any storage class, including NFS and local-path.

My storage is NFS (nfs-client from a Proxmox LXC) and local-path. Neither has a CSI snapshot provider. Without filesystem backup, Velero has no way to capture PVC contents — it can only back up the Kubernetes resource definitions (the PV and PVC objects themselves, not the data inside them).

The default is snapshot-based. If you don't explicitly tell Velero to use filesystem backup, it takes snapshots. If no snapshot provider exists, it takes no snapshot and the backup "succeeds" with only manifests.

The Discovery

The backup schedule was:

# kubernetes/system/velero/schedule.yml
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily-backup
  namespace: velero
spec:
  schedule: "0 5 * * *"
  template:
    includedNamespaces: ["apps", "vault", "database", "argocd"]
    excludedResources: ["events", "events.events.k8s.io"]
    ttl: 720h

No defaultVolumesToFsBackup. No per-backup volume hooks. No snapshot provider configured. Every backup since the schedule was created had captured only Kubernetes manifests.

The size of the backups was misleading. Manifests for 25+ applications, their Services, ConfigMaps, Secrets, and ExternalSecrets add up to a few MB — close enough to what I expected for a "metadata-only" backup that I didn't question it.

The Fix

# kubernetes/system/velero/schedule.yml
spec:
  template:
    defaultVolumesToFsBackup: true
    includedNamespaces: ["apps", "vault", "database", "argocd"]

One line. defaultVolumesToFsBackup: true tells Velero to use Kopia for filesystem backup of every PVC in every included namespace. Kopia runs as a sidecar, mounts the PVC, and copies the files to the Garage S3 bucket.

After adding this flag, the next backup jumped from ~2 MB to ~4 GB — the actual data.

Verification

The real lesson: velero backup describe with --details is the only way to verify that PVC data was actually captured:

velero backup describe daily-backup-20260620050012 --details | grep -A5 "Pod Volume Backups"
# Phase: Completed
# Total items: 342
# ...
# Pod Volume Backups:
#   Completed: 12
#   Failed: 0

If "Pod Volume Backups" is missing or shows 0 completed, no filesystem backup happened. The backup "completed" but only captured manifests. The velero backup describe output is the only place this distinction is visible — velero backup get just shows Phase: Completed regardless.

For automated verification, I added a post-backup check:

# After each backup completes
COMPLETED=$(velero backup describe $BACKUP --details -o json | \
  jq '.status.progress_podVolumeBackups | .completed // 0')
if [ "$COMPLETED" -eq 0 ]; then
  echo "WARNING: No pod volume backups captured"
  # Alert via Discord webhook
fi

The Broader Problem: Backup Circularity

The discovery of the false-positive backup led to a deeper issue documented in docs/garage-velero-design-2026-07-17.md: Velero backs up to Garage S3, which runs inside the same cluster. If the cluster dies, both Velero and Garage are gone — the backup target is inside the thing being backed up.

This is the classic backup circularity problem:

Velero (k8s) → Garage S3 (k8s) → same cluster

The backup is only useful for recovering individual PVCs or namespaces within a running cluster. For full cluster recovery, you need the backup to exist outside the cluster — which is why the Cloudflare R2 offsite backup scaffolding exists but isn't active yet.

The secondary issue is "torn state" — Velero's metadata (manifests) and the PVC filesystem backups are stored separately. If one succeeds and the other fails, you can end up with manifests but no data, or data but no manifests to restore it into. The defaultVolumesToFsBackup flag doesn't solve torn state, but it at least ensures both parts of the backup are attempted.

What Would Have Caught This

A restore test. Running velero restore against a test namespace monthly would have caught the empty PVCs within the first cycle. The backup "Completed" status gave false confidence.
Backup size monitoring. A 2 MB backup for a cluster with 25+ apps and 4 GB of PVC data is obviously wrong. An alert on backup size below a threshold would have caught it.
Post-backup verification. The velero backup describe --details | grep "Pod Volume Backups" check should run after every scheduled backup, not just when you need a restore.

Backup verification is the same problem in Azure: Azure Backup reports "Completed" for VM snapshots, but a snapshot without the correct recovery point tier doesn't include disk contents. The verification step — actually restoring a test VM from backup — is the only way to confirm the backup contains what you think it contains. Compliance frameworks like ISO 27001 and NIS2 require documented restore testing for exactly this reason.

Self-Hosted SSO for 25 Services: Authelia OIDC on Kubernetes

david — Sat, 08 Aug 2026 15:08:56 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Every internal service in my homelab goes through the same authentication gate: Authelia. Proxmox, PBS, Grafana, ArgoCD, Headscale, ArgoCD, Uptime Kuma, Paperless, Nextcloud — 25+ web services, one login, one session, one set of access rules. The OIDC provider, the Postgres backend, the session store, and the secrets are all running inside k3s, backed by CNPG, Redis, and Vault.

This article is the full implementation: how the pieces fit together, why certain design decisions were made, and the specific bugs that bit me along the way.

View the complete homelab infrastructure source on GitHub 🐙

The Architecture

Authelia runs as a Kubernetes Deployment in the apps namespace, protected by the same default-deny NetworkPolicy that applies to everything else. It has three dependencies:

PostgreSQL — CNPG-managed postgres-authelia cluster in the database namespace
Redis — session store, ephemeral (no persistence needed)
Vault — hmac_secret, OIDC private keys, JWT secrets, session secrets

The Traefik ForwardAuth middleware sits in front of every service. When a request hits Traefik, the middleware sends a verification request to Authelia's /api/verify endpoint. Authelia checks the session cookie, validates the OIDC token if applicable, and returns a 200 (allowed) or 401 (redirect to login).

# kubernetes/apps/authelia/middleware.yml
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: authelia
  namespace: apps
spec:
  forwardAuth:
    address: "http://authelia.apps.svc.cluster.local:9999/api/verify"
    trustForwardHeader: true
    authResponseHeaders:
      - Remote-User
      - Remote-Groups
      - Remote-Email

Every IngressRoute that needs protection adds middlewares: [{name: authelia}]. Services that need API-level protection (not browser-based) use OIDC client credentials instead.

The OIDC Configuration

Authelia acts as an OIDC provider for services that support it. The ConfigMap defines five OIDC clients:

# kubernetes/apps/authelia/configmap.yml
identity_providers:
  oidc:
    clients:
      - id: proxmox
        description: Proxmox VE
        secret: <from vault>
        authorization_policy: two_factor
        scopes: ["openid", "profile", "email"]
        redirect_uris: ["https://pve.woitzik.dev:8006/pam2/callback/oidc"]
      - id: pbs
        description: Proxmox Backup Server
        secret: <from vault>
        authorization_policy: two_factor
        redirect_uris: ["https://pbs.woitzik.dev:8007/pam2/callback/oidc"]
      - id: argocd
        description: ArgoCD
        secret: <from vault>
        authorization_policy: two_factor
        grant_types: ["authorization_code"]
        redirect_uris: ["https://argo.woitzik.dev/auth/callback"]
      - id: grafana
        description: Grafana
        secret: <from vault>
        authorization_policy: two_factor
        redirect_uris: ["https://monitoring.woitzik.dev/login/generic_oauth"]
      - id: headscale
        description: Headscale
        secret: <from vault>
        authorization_policy: two_factor
        redirect_uris: ["https://headscale.woitzik.dev/oauth2/callback"]

Key detail: ArgoCD uses client_secret_post for token exchange, while all other clients use client_secret_basic. This is a quirk of ArgoCD's OIDC implementation — it sends the client secret in the POST body rather than the Authorization header.

Secrets in Vault

The OIDC hmac_secret, signing keys, and session secrets were originally committed as plain Kubernetes Secrets. This is fine for a homelab, but it means anyone with kubectl get secret can read them.

The migration to Vault:

# kubernetes/apps/authelia/external-secret.yml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: authelia-secrets
  namespace: apps
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: authelia-secrets
    creationPolicy: Merge
  data:
    - secretKey: hmac-secret
      remoteRef:
        key: secret/authelia
        property: hmac-secret
    - secretKey: oidc-issuer-private-key
      remoteRef:
        key: secret/authelia
        property: oidc-issuer-private-key
    - secretKey: jwt-secret
      remoteRef:
        key: secret/authelia
        property: jwt-secret
    - secretKey: session-secret
      remoteRef:
        key: secret/authelia
        property: session-secret
    - secretKey: redis-password
      remoteRef:
        key: secret/authelia
        property: redis-password

creationPolicy: Merge means the ExternalSecret creates the Kubernetes Secret if it doesn't exist, then updates it on each refresh cycle. If Vault is sealed or unreachable, the existing Secret persists — Authelia can continue operating with stale secrets until the next refresh.

The users database (username, argon2id hash, group memberships) was also migrated from a plain Secret to an ExternalSecret sourced from Vault. This one required a creationPolicy: Merge approach because the file is referenced by path in the Authelia config, not as an environment variable.

The Postgres Backend

CNPG manages the postgres-authelia single-instance cluster in the database namespace. WAL archiving goes to Garage S3 for point-in-time recovery:

# postgres cluster config
postgresql:
  parameters:
    max_connections: "100"
    shared_buffers: "256MB"
    effective_cache_size: "512MB"
backup:
  barmanObjectStore:
    destinationPath: "s3://postgres-backups"
    endpointURL: "https://s3.woitzik.dev"
  retentionPolicy: "30d"

Daily ScheduledBackup resources create full base backups. The combination of WAL archiving + daily base backups gives PITR granularity down to the transaction level.

The Authelia Schema Bug

One gotcha: Authelia's Postgres schema version can outpace the running image version. If Vault or Renovate bumps the Authelia image while the database has already been migrated to a newer schema, Authelia starts with a schema mismatch and fails.

The fix was straightforward — bump the image version to match the schema:

image: ghcr.io/authelia/authelia:4.39.20  # matches DB schema v24

But the symptom was confusing: Authelia reported healthy in ArgoCD, the pods were running, but login attempts returned 500 errors. The health check endpoint (/api/health) doesn't validate database schema compatibility — it only checks that the process is up and can reach Postgres.

Access Control Rules

The access_control section defines who can access what:

access_control:
  rules:
    - domain: auth.woitzik.dev
      policy: bypass
    - domain: "*.woitzik.dev"
      policy: two_factor

Critical detail: auth.woitzik.dev must be the first rule. Authelia matches rules top-down and stops at the first match. If the wildcard *.woitzik.dev → two_factor rule comes before the auth.woitzik.dev → bypass rule, Authelia redirects to itself — creating an infinite redirect loop.

I hit this bug during initial setup. The logs showed repeated 302 redirects between auth.woitzik.dev and itself, which looks like a configuration error but is actually a rule ordering problem.

The Two-Replica Setup

Authelia runs at two replicas with a PodDisruptionBudget (minAvailable: 1). Redis handles session state, so either replica can serve any session. The only requirement is that both replicas share the same hmac_secret and JWT signing keys — which they do, since both read from the same Vault-backed ExternalSecret.

# kubernetes/apps/authelia/authelia.yml
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: authelia
          livenessProbe:
            httpGet:
              path: /api/health
              port: 9999
          readinessProbe:
            httpGet:
              path: /api/health
              port: 9999

The health probes check /api/health, which validates Postgres connectivity, Redis availability, and configuration file integrity. Blackbox Exporter probes the same endpoint externally — catching cases where Traefik returns 200 but Authelia itself is down (the Traefik → Authelia middleware can return 200 on connection failure if not configured carefully).

What I'd Change

Two things, if starting over:

Start with Vault from day one. Migrating secrets from plain Kubernetes Secrets to Vault-backed ExternalSecrets after the fact required a creationPolicy: Merge dance that wouldn't have been necessary if the secrets were never in git.
Use client_secret_post for everything. ArgoCD's client_secret_post quirk means I can't use a single client template — every OIDC client needs its own configuration. If all clients used the same grant type, the ConfigMap would be simpler.

SSO across 25+ services is the same problem in enterprise Azure: Entra ID provides the OIDC provider, Azure Application Registrations replace the Authelia client definitions, and Conditional Access Policies replace the access_control rules. The scale is different (Entra ID handles millions of identities), but the architectural pattern — centralize authentication, delegate authorization to the service, protect everything behind a single middleware — is identical.

My Terraform Runner Destroyed Itself Mid-Apply

david — Mon, 03 Aug 2026 09:54:57 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

On 2026-07-04, Atlantis applied a Terraform change that shut down the VM it was running on. The Terraform runner destroyed its own runtime environment mid-apply.

This is the story of a circular dependency that doesn't show up in terraform plan, why bpg/proxmox can't always update attributes in-place, and how moving Atlantis from a k3s Deployment to a dedicated LXC container eliminated the hazard entirely.

View the complete homelab infrastructure source on GitHub 🐙

The Setup

Atlantis was deployed as a k3s Deployment in the apps namespace, running on one of three k3s VMs managed by the same Proxmox Terraform stack. The Proxmox Terraform configuration (terraform/stacks/proxmox/) defines all three k3s VMs, their CPU, memory, disk, and boot settings.

The circular dependency: Atlantis runs on k3s VMs → Terraform manages k3s VMs → Atlantis applies Terraform changes to k3s VMs.

In practice, this was safe for most changes. memory, disk.size, cpu.cores — these can all be updated in-place by bpg/proxmox without stopping the VM. But certain attributes require a full VM shutdown-and-restart cycle. Specifically: cpu.units (scheduling priority) and serial_device configuration.

The Kill Shot

The Terraform diff for a cpu.units change looks like this:

resource "proxmox_virtual_machine" "vm_srv_k3s_11" {
  # ...
  cpu {
    units = 2048  # 2x scheduling priority over LXCs
  }
}

When bpg/proxmox detects a change to cpu.units, it can't hot-apply it. The provider issues a qmshutdown via the Proxmox API, waits for the VM to stop, applies the change, then starts the VM again.

On 2026-07-04, a PR changed cpu.units on one of the k3s VMs. Atlantis picked up the PR, ran terraform plan, showed the diff (in-place update), and ran terraform apply. The bpg/proxmox provider sent qmshutdown to the Proxmox API for the VM running the Atlantis pod.

The VM shut down. The Atlantis pod was killed. The Terraform apply was interrupted mid-execution. Kubernetes rescheduled the pod on a different node, but the apply state was lost.

The same thing happened again on a serial_device attribute change — another attribute that requires a VM shutdown. Two occurrences, same root cause: the Terraform runner was managing the infrastructure it depended on for its own execution.

Why `terraform plan` Doesn't Catch This

The circular dependency isn't expressed in the Terraform configuration. Atlantis's Pod spec doesn't reference the Proxmox VMs, and the Proxmox VMs don't reference Atlantis. Terraform sees two independent resource graphs. The dependency is physical, not declarative — Atlantis runs on the VMs, but Terraform doesn't know that.

terraform plan shows "will update in-place" for cpu.units. It doesn't know that "update in-place" means "shut down the VM first." That behavior is a provider implementation detail, not something Terraform's planning phase understands.

The lifecycle.ignore_changes block can prevent specific attributes from being planned, but that's a workaround, not a fix. You'd be ignoring a real change to avoid a structural hazard.

The Fix: Move Atlantis Off k3s

The fix was ADR-012: move Atlantis from a k3s Deployment to a dedicated LXC container (ct-srv-atlantis-01, VMID 204, IP 10.0.20.250).

# terraform/stacks/proxmox/lxc.tf
resource "proxmox_virtual_machine" "ct_srv_atlantis_01" {
  # Dedicated LXC — NOT managed by the same Terraform stack
  # Atlantis manages OTHER stacks, but its own container is outside the scope
  vm_id   = 204
  name    = "ct-srv-atlantis-01"
  node_name = "pve"
  # ...
}

The key difference: the Atlantis LXC is still defined in the Proxmox Terraform stack, but it's never managed by Atlantis itself. The atlantis.yaml repo config whitelists only specific repos and directories:

repos:
  - name: github.com/dwoitzik/homelab-infrastructure
    allowed_overrides: [apply_requirements, delete_source_branch_on_merge]
    apply_requirements: [approved, mergeable]
    projects:
      - dir: terraform/stacks/network
        workspace: default
      - dir: terraform/stacks/cloudflare
        workspace: default
      - dir: terraform/stacks/garage
        workspace: default

Notice: terraform/stacks/proxmox/ is not listed. Atlantis can plan and apply network, cloudflare, and garage changes — but never Proxmox changes. Proxmox changes go through a separate review process, or I apply them manually after careful review.

This breaks the circular dependency structurally: Atlantis manages everything except the infrastructure it runs on.

The Ansible Layer

The Atlantis LXC runs via Ansible, not k3s:

# ansible/roles/atlantis/tasks/main.yml
- name: Deploy Atlantis via Docker Compose
  community.docker.docker_compose_v2:
    project_src: /opt/atlantis
    state: present

The custom Dockerfile includes the Proxmox self-signed CA cert (pve-root-ca.crt) so Atlantis can talk to the Proxmox API over HTTPS:

FROM ghcr.io/runatlantis/atlantis:v0.30.0
COPY pve-root-ca.crt /usr/local/share/ca-certificates/pve-root-ca.crt
RUN update-ca-certificates

The Proxmox API token, Cloudflare API token, and MikroTik credentials are all stored in Ansible Vault and injected via Docker Compose environment variables.

The Pattern

Any Terraform runner that manages the infrastructure it runs on has this hazard. In a cloud environment, it's less obvious because terraform apply against an Azure VM doesn't restart the VM — Azure handles in-place updates at the platform level. But the same structural dependency exists: an Atlantis instance running on an Azure VM that manages that VM's NSG, disk, or network interface.

The clean fix is always the same: the runner manages everything except itself. If that's not possible, lifecycle { ignore_changes } on attributes that trigger restarts is the minimum viable mitigation.

The same pattern applies to CI/CD runners in enterprise environments. A self-hosted GitHub Actions runner managing its own host's infrastructure via Terraform has the identical circular dependency. The fix is the same: separate the runner's infrastructure from the infrastructure it manages, even if they share the same cloud account.

The ZFS ARC Freeze: How a Marginal PSU Killed My Entire Homelab

david — Sat, 01 Aug 2026 00:55:59 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Three times in one week, my Proxmox host froze completely. No SSH, no web UI, no console. The only recovery was a hard power cycle. Each time, I lost every running VM and LXC simultaneously — k3s control plane, workers, databases, all of it.

The first time, I blamed a kernel bug. The second time, I blamed ZFS. The third time, I found the actual chain: a marginal PSU causing CPU throttling under load, which triggered a ZFS 2.4.1 ARC deadlock that locked the entire host.

View the complete homelab infrastructure source on GitHub 🐙

The Symptoms

The freezes weren't random. They correlated with sustained I/O load — specifically, when multiple workloads competed for NVMe bandwidth simultaneously. The pattern:

k3s etcd writing to the same NVMe as the ZFS pool
Ollama LLM inference spinning up large model loads
Paperless OCR bursts hitting the disk queue hard

Under moderate load, everything was fine. Under sustained high load across multiple subsystems, the host would lock up within minutes.

The first two freezes produced no useful logs. journalctl cut off mid-entry. No kernel panic, no oops, no MCE. Just silence.

The PSU Theory

The BMAX Mini PC* ships with a 90W PSU. Under normal operation, the Ryzen 7 5825U draws 15W TDP and the NVMe pulls another 8W. But when all cores are loaded — k3s workers, Ollama inference, OCR workers — the CPU can boost well above base TDP.

I added power monitoring via a TP-Link Tapo P110* to track actual draw:

Idle:              47W
k3s + normal apps: 68W
All cores loaded:  94W
Peak (all + NVMe): 108W

108W peak against a 90W PSU. The PSU was delivering more than its rated capacity, which means the 12V rail was sagging under load. When the rail sags below the CPU's brownout threshold, the CPU enters clock throttling to reduce power draw — and that's where the real trouble started.

The ZFS Deadlock

ZFS uses a Transaction Group (txg) commit cycle. Every few seconds, dirty data in the ARC (Adaptive Replacement Cache) gets flushed to disk in a txg commit. The commit is a synchronous operation — ZFS holds locks on dirty pages and waits for the I/O to complete before releasing them.

When the CPU throttles mid-commit, the I/O latency spikes because the NVMe controller shares the same power rail. The txg commit can't complete within its timeout window, ZFS holds the ARC locks, new I/O requests pile up waiting for those locks, and the entire storage stack deadlocks.

In ZFS 2.4.1 specifically, the ARC eviction path has a known issue where it can block on the same locks held by a stalled txg commit. This creates a circular wait: ARC eviction needs txg to complete, txg needs I/O bandwidth, I/O is delayed by CPU throttling, CPU throttling is caused by the PSU sag, and the PSU sags because everything is running at once.

The host doesn't panic. It just stops responding to anything that touches disk.

The Fixes

Fix 1: Disk I/O Isolation

The k3s VMs and the ZFS pool were sharing the same NVMe. QEMU's page cache was adding to the I/O contention. I moved the k3s VM disks to cache=none + aio=native:

# terraform/stacks/proxmox/vm.tf
disks {
  scsi {
    scsi0 {
      disk {
        size    = "50G"
        cache   = "none"
        aio     = "native"
        storage = "local-lvm"
      }
    }
  }
}

cache=none bypasses QEMU's page cache entirely — the guest handles its own caching. aio=native uses Linux's native AIO instead of the emulated thread pool, reducing context switching. This alone reduced I/O latency variance by 40% under load.

Fix 2: ZFS ARC Limits

Without caps, the ARC can grow to consume most of available memory, which means more dirty pages waiting for txg commit. I capped the ARC:

# /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=4294967296  # 4GB max ARC

On a 64GB host, 4GB for ARC is generous enough for read caching without creating a massive pool of dirty pages that can deadlock under I/O pressure.

Fix 3: txg Timeout Tuning

ZFS defaults to a 5-second txg timeout. On hardware with known I/O fragility, I shortened it:

# /etc/modprobe.d/zfs.conf
options zfs zfs_txg_timeout=2

A shorter timeout means ZFS gives up on a stalled commit sooner, which sounds counterintuitive, but it prevents the deadlock from propagating. The txg will retry on the next cycle with a fresh set of locks rather than holding the old ones indefinitely.

Fix 4: Staggered Boot Order

The boot storm was a separate but related trigger. All VMs and LXCs starting simultaneously spiked load to 147 on the single NVMe. I added staggered boot ordering:

# NFS boots first (no k3s dependency)
# k3s VMs boot 30s apart
# LXCs start last

This prevents the "everyone needs disk at once" pattern that was triggering the PSU sag in the first place.

What It Taught Me

The root cause wasn't software — it was hardware. But the software (ZFS 2.4.1) turned a power delivery problem into a host-level deadlock. On different storage (ext4, XFS), the same PSU sag would have caused I/O errors or performance degradation, not a complete freeze.

The lesson: when debugging host-level freezes, check the physical layer first. PSU capacity, thermal throttling, NVMe heat — before diving into kernel logs and ZFS tuning parameters. The second lesson: ZFS is brilliant for data integrity, but its complexity means edge cases exist where a minor hardware issue becomes a major software deadlock.

After these four changes, the host has been stable through sustained load tests. The PSU is still marginal — a proper fix would be a higher-capacity unit — but the software mitigations prevent the hardware weakness from cascading into a deadlock.

Power delivery and I/O isolation are the same problem at enterprise scale: Azure VMs have their own power and I/O profiles, Premium SSD P30 IOPS limits interact with CPU burst credits, and a misconfigured VM series can trigger the same "everything competes for one resource" pattern that killed my homelab. The difference is that Azure hides the PSU behind an SLA — you just get a throttled VM instead of a frozen host.

How a Single Volume Was 65% of My Velero Backup and What I Almost Excluded Instead

david — Mon, 27 Jul 2026 10:25:30 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Velero's nightly backup was taking 207 seconds and backing up 33.4GB. That's normal for a cluster with 15+ PVCs — except when one volume accounts for 65% of the total bytes. The nfs-provisioner's root-mount volume was backing up the entire shared NFS export as a single blob, even though every app's individual PVC (a subdirectory of the same tree) was already backed up separately.

The fix was a one-line annotation. The investigation almost went wrong when the first hypothesis pointed at the wrong disk entirely.

The Symptom

Velero's backup stats showed a disproportionate volume:

PodVolumeBackup: nfs-provisioner-root
Namespace: nfs-provisioner
Size: 33.4GB
Duration: 207s

Total backup for that run: ~51GB. This single volume was 65% of the total. Every other PVC was 1-4GB.

The Investigation

The nfs-provisioner pod mounts the entire shared NFS export at /persistentvolumes. Every app's PVC is a subdirectory of this tree. Velero's defaultVolumesToFsBackup: true setting means every volume the pod mounts gets backed up — including the root mount that contains all the subdirectories.

Each app's own pod also mounts its PVC subdirectory separately. Velero backs that up too. So the same data is backed up twice: once as part of the root mount, once as the individual PVC.

The First Hypothesis (Wrong)

My first guess was garage-data/immich-library — Immich's photo library, which lives on separate USB-attached disks (Seagate Portable 2TB External HDD*). Excluding it would have reduced backup size, but the USB disks aren't on the contended rpool. The backup bottleneck was the SSD, not USB. Excluding the Immich volume would have been the wrong fix for the right symptom.

The Right Fix

Verified the target before acting:

# Confirm the root-mount volume targets rpool (the contended disk)
findmnt /persistentvolumes
# /dev/zvol/rpool/nfs  ← confirmed rpool

# Confirm the individual PVCs are subdirectories of the same tree
ls /persistentvolumes/
# default-elasticsearch-data-0/
# default-immich-redis-data-0/
# default-minecraft-data/
# ... (each is a separate PVC)

The root mount is the entire rpool. Every individual PVC is a subdirectory. Backing up the root mount redundantly backs up everything.

The Exclusion

# kubernetes/system/nfs-provisioner/application.yml
podAnnotations:
  backup.velero.io/backup-volumes-excludes: nfs-subdir-external-provisioner-root

One annotation. Velero skips the root-mount volume entirely. Every app's individual PVC still gets backed up through its own pod's mount.

Result

Before: 51GB total, 33.4GB from root mount (65%)
After:  17.6GB total, 0GB from root mount

Backup duration dropped from 207s to ~60s. No data was lost — every PVC is still backed up individually.

The Pattern

This is a common Velero gotcha with NFS provisioners:

NFS provisioner pod mounts the entire export root
App pods mount individual PVC subdirectories
Velero backs up both — root mount AND individual PVCs
The root mount is redundant because it contains everything the individual PVCs contain

The fix is always the same: exclude the root mount via backup.velero.io/backup-volumes-excludes.

How to Find Redundant Backups

# List all PodVolumeBackups and their sizes
kubectl get podvolumebacks -A -o json | \
  jq -r '.items[] | "\(.spec.volume) \(.status.progress.totalBytes // 0) \(.metadata.namespace)"' | \
  sort -k2 -rn | head -10

If one volume is disproportionately large and its data is already covered by individual PVC backups, it's redundant.

🚀 Azure Firewall - Enterprise Forced Tunneling Edition — €49

Cycle-error-free resource ordering - deploys first time, every time
KMS & Azure AD bypass routes - no broken Windows VMs or auth failures
Dynamic for_each subnet binding - scales to any number of Spokes
IP Group-based firewall policies - no hardcoded IP addresses
FQDN baseline rules for Windows Updates and core Microsoft services

Get the enterprise module →

Full source code · one-time payment · instant download

Velero's defaultVolumesToFsBackup: true is convenient but dangerous with NFS provisioners. The root mount contains everything, and Velero doesn't know it's redundant. Check your PodVolumeBackup stats — if one volume is 65% of your total, you're backing up the same data twice.

Why a Cloud Backup Sync Was Failing Daily for Six Weeks

david — Mon, 27 Jul 2026 10:08:19 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

A systematic ansible-playbook --check --diff sweep across every host group turned up a stale Vault credential and a cron job that appeared to have simply stopped running. The initial read was a straightforward drift problem: fix the credential, restore the cron schedule, move on. That read was wrong on both the diagnosis and, as it turned out one PR later, the intended fix.

View the complete homelab infrastructure source on GitHub 🐙

What the Audit Sweep Found

Two issues on the PBS management host, both real:

1. A stale Vault credential that would have clobbered a working live token. The pbs_rclone_gdrive_token stored in Vault was from April; the live rclone.conf on the host had a June token - rclone had refreshed it itself as OAuth tokens do, and Vault had never been updated to match. Running the Ansible role for real (not just --check) would have overwritten the working, self-refreshed credential with a four-month-stale one, breaking authentication.

# Confirmed: live rclone.conf token newer than Vault's stored value
diff <(rclone config show pbs-gdrive | grep token) \
     <(vault kv get -field=pbs_rclone_gdrive_token secret/pbs)
# Fixed: update Vault to match the live, working token
vault kv patch secret/pbs pbs_rclone_gdrive_token="$(rclone config show pbs-gdrive | grep token)"

2. The offsite sync cron job simply wasn't on the host. Its own log file, pbs-to-gdrive.log, showed it running - mostly failing - from May 4 through June 14, then nothing. No entries at all after that date. Something had stopped the schedule.

The Wrong First Conclusion

The natural read of "cron job disappeared, logs show it was mostly failing before that" is: something broke it, and the failures were probably why. Re-enable the schedule, fix whatever was causing the failures, done.

Actually diagnosing the failures required running a real, bounded manual sync rather than reasoning from the log lines alone:

rclone sync /mnt/pbs-datastore gdrive:pbs-backup --transfers 4 --checkers 8 \
  --stats 30s --stats-one-line --max-duration 30m
# Transferred: 3.2 MiB / ~2.1 TiB, 0%, 1.6 KiB/s, ETA 12 weeks

1.6 KiB/s. Not a credential failure, not a network failure - Google Drive's API was throttling the sync hard, and the reason became clear from what PBS actually stores on disk: not a handful of large archive files, but tens of thousands of small deduplicated chunk files. Google Drive's API rate-limits per-file operations aggressively, and a chunk-based datastore is close to the worst-case access pattern for that limit.

At an observed throughput of ~1.6 KiB/s against a multi-terabyte datastore, the estimated time for just the initial full sync was on the order of 12 weeks. The daily cron job, scoped to a 24-hour window, could never have completed a first full sync - every single run for six weeks was doomed from the start, not by a bug in the job but by a structural mismatch between the storage format and the destination's API characteristics.

The Fix That Turned Out to Be Wrong

The first PR treated the missing cron job as accidental drift and restored it - re-added the scheduled task, assuming it had been silently dropped by some unrelated config change and needed putting back.

It hadn't been silently dropped. Checking with the account owner surfaced the actual story: the cron job had been turned off deliberately, because the destination Google Drive account doesn't have enough free storage quota for the PBS datastore's full size. Turning the schedule back on wasn't fixing drift - it was un-doing an intentional decision that the earlier PR had mistaken for an accident.

# What the first PR did (wrong - re-enables a deliberately-disabled job)
- name: Sync PBS offsite cron
  ansible.builtin.cron:
    name: pbs-to-gdrive-sync
    state: present   # ← reintroduces a schedule the user turned off

# The corrected version
- name: Remove PBS offsite cron (deliberately disabled -- insufficient quota)
  ansible.builtin.cron:
    name: pbs-to-gdrive-sync
    state: absent

The correction PR also removed the live crontab entry that the first "fix" had reintroduced, and rewrote the documentation to describe the situation accurately: not "broken, needs re-enabling" but "deliberately deferred pending a decision about storage quota or a different offsite strategy."

The Same Log Pattern, Two Opposite Conclusions

The interesting part isn't the throttling finding by itself - it's that the same evidence (pbs-to-gdrive.log showing failures, then silence) supported two completely different, mutually exclusive conclusions:

"This is drift - something broke, restore the intended state."
"This is intentional - a human made a deliberate decision that the automation shouldn't override."

Ansible's entire idea of "correct state" assumes the desired state is knowable from the playbook. It has no way to distinguish "this differs from the playbook because of an accident" from "this differs from the playbook because a human changed their mind after the playbook was written." Both look identical in a --diff output. The only way to tell them apart is to ask.

This is also, notably, the same false-positive class that closed out a side finding in the same PR: an AdGuard admin-password "drift" flagged by the same sweep turned out to be bcrypt's per-render random salt making two hashes of the same password look different in a diff - not a real config drift either. Two findings from one audit sweep, both initially read as drift, both actually non-issues once verified against ground truth rather than assumed from the diff output.

What This Argues For

Any Ansible task that can toggle a schedule, feature flag, or setting on or off needs a documented reason when it's off - not just an inferred one from "the playbook says it should be on." A short comment (# disabled 2026-06-XX: destination quota insufficient, see docs/backup-strategy.md) turns a future --diff output from "ambiguous, ask the owner" into "documented, already explained."

And more broadly: a drift-detection sweep is good at finding that something differs from the intended state. It says nothing about why it differs, and treating every diff as an accident to be corrected is exactly as wrong as treating every diff as intentional and ignoring it. Both directions of this mistake happened in the same PR chain here, within hours of each other.

The direct Azure parallel: Azure Policy remediation tasks that "fix" a resource back to policy-compliant state can just as easily undo a deliberate, documented exception a team made for a legitimate reason. Policy exemptions exist for exactly this - but only if someone actually creates the exemption object instead of leaving the deviation undocumented and hoping nobody re-applies the policy. An undocumented intentional deviation and an actual accidental drift look identical to any automated compliance sweep.

MikroTik vs Ubiquiti for Home Network 2026

david — Sun, 19 Jul 2026 11:07:31 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

View the complete homelab infrastructure source on GitHub 🐙

This is the networking debate that never dies in homelab circles: MikroTik or Ubiquiti? I have run both in production — MikroTik RB5009 as my primary router for over two years with every firewall rule and VLAN managed via Terraform, and Ubiquiti UniFi gear in a friend's lab where I helped with deployment. This is not a spec-sheet comparison. It is a practical breakdown of what each platform actually delivers for a homelab that runs Proxmox, K3s, and zero-trust networking.

TL;DR — The Verdict

MikroTik RB5009 if you want maximum control, Terraform-native automation, and the best price-to-performance in router hardware. The learning curve is steep, but the ceiling is infinitely higher.

Ubiquiti UniFi Dream Router if you want plug-and-play setup, a polished UI, and an ecosystem where access points, switches, and cameras all manage from one dashboard. You pay a premium and give up depth of control.

The short version: MikroTik is for engineers who want to own their network stack. Ubiquiti is for users who want a network that works without deep networking knowledge. Both are valid — but for a homelab running infrastructure-as-code, MikroTik wins decisively.

Quick Comparison

Feature	MikroTik RB5009	Ubiquiti UniFi Dream Router
Price	~€180*	~€300*
CPU	Marvell 98DX3236 (ARM)	Quad-core ARM Cortex-A57
RAM	1GB	2GB
Ports	7x GbE + 1x 10G SFP+	4x GbE + 1x 2.5GbE WAN
Switch Chip	Yes (hardware offloading)	Integrated (software)
OS	RouterOS 7	UniFi Network
Management	CLI, WinBox, API, Terraform	UniFi Controller (UI)
VLAN Support	Full (bridge VLAN filtering)	Full (VLAN networks)
VPN	WireGuard, OpenVPN, IPsec, PPTP	WireGuard, OpenVPN, L2TP
Wi-Fi	None (bring your own AP)	Built-in Wi-Fi 6
Price/Performance	Excellent	Moderate

Category-by-Category Breakdown

Design & Build Quality

The RB5009 is an industrial-looking metal box. No Wi-Fi, no antennas, no RGB. It has 7 Gigabit Ethernet ports, one 10G SFP+ cage, and a USB 3.0 port. The metal chassis dissipates heat passively — no fan, no noise. It is designed to live in a rack or on a shelf without anyone noticing it. At 100x155x33mm, it is compact enough to mount in a 10-inch rack (I use a DIGITUS 10" rack* for mine).

The UniFi Dream Router (UDR) is designed to be seen. It is a white, cylindrical unit with a built-in Wi-Fi 6 access point and a small LED status ring. It looks like something from a modern living room, not a server closet. The build quality is good — solid plastic, well-ventilated — but it is designed for desk or shelf placement, not rack mounting. The form factor says "consumer product" while the RB5009 says "networking equipment."

Winner: MikroTik RB5009 for homelab use. The fanless design, rack-friendly form factor, and 10G SFP+ port make it the better infrastructure component. The UDR wins for living room aesthetics if that matters to you.

Features & Flexibility

This is where the gap becomes a canyon.

RouterOS 7 on the RB5009 is, functionally, a full Linux networking stack exposed through a proprietary CLI and API. You get: bridging with VLAN filtering, firewall with stateful packet inspection, NAT (source and destination), QoS (HTSQ, PCQ, CAKE), MPLS, BGP, OSPF, VPLS, bonding, MLAG, traffic shaping, bandwidth limiting, DHCP server with static bindings, DNS forwarding with split-horizon, CAPsMAN (centralized AP management), and more. The feature set is closer to enterprise networking gear (Cisco, Juniper) than to consumer routers.

The UDR runs UniFi Network, which is a software-defined networking platform. It provides: VLAN segmentation, firewall rules, IDS/IPS (Intrusion Detection/Prevention), traffic identification, guest portal, and DPI (Deep Packet Inspection). The UniFi Controller — a web application that runs either on the UDR itself or on a separate server — manages all of it through a graphical interface.

The UniFi feature set is impressive for what it is, but it is fundamentally a curated subset. You get the features Ubiquiti decided to implement, configured the way Ubiquiti decided to configure them. MikroTik gives you the entire toolbox and lets you build whatever you want. For a homelab that runs Terraform-managed infrastructure, the difference is critical — MikroTik's API and CLI let you automate everything; UniFi's API is limited and unofficial.

Winner: MikroTik RB5009 for depth and automation potential. UniFi UDR for breadth of managed features out of the box (IDS/IPS, DPI, traffic identification).

Ease of Use

Ubiquiti wins this category flatly, and it is not close.

The UniFi Controller provides a visual dashboard showing network topology, connected clients, traffic flows, and security events. VLAN creation is a form fill. Firewall rules are a guided wizard. Firmware updates are one-click. If you have never configured a router before, the UDR will have your network segmented and secured in under 30 minutes.

RouterOS is a different world. The WinBox GUI exists and works, but it mirrors the CLI structure — you navigate menus organized by protocol and feature, not by task. Creating a VLAN involves: creating a bridge, adding ports to the bridge, creating VLAN entries on the bridge, creating an interface for the VLAN, assigning an IP address, creating a DHCP server, and creating firewall rules for the new network. Each step is a separate resource. The CLI (or Terraform) makes this repeatable; the GUI makes it tedious.

I will not sugarcoat it: RouterOS has a steeper learning curve than any consumer router OS. The first time you configure Bridge VLAN Filtering, you will lock yourself out. The second time, you will not. By the third time, you will understand L2 networking better than 99% of home network users. That knowledge has value beyond MikroTik — it transfers to every networking platform.

Winner: Ubiquiti UDR for initial setup and ongoing management. MikroTik for users who want to actually understand their network.

Security & Firewall

Both platforms support stateful packet inspection, NAT, and rule-based firewalling. The differences are in philosophy and depth.

MikroTik gives you full control over every firewall rule. The RouterOS firewall is a chain-based system (input, forward, output) with rule ordering that matters. You can filter by source/destination IP, port, protocol, connection state, interface, VLAN, packet content, and more. The zero-trust approach I documented in MikroTik Zero Trust Firewall with Terraform is built entirely on this: default-drop on input and forward, with explicit allow rules for each service.

Here is the actual Terraform code that creates the final drop rule in my firewall:

resource "routeros_ip_firewall_filter" "fwd_99_drop_all" {
  action  = "drop"
  chain   = "forward"
  comment = "99: Global - Final Drop (Zero Trust Policy)"
}

That single rule, placed last in the chain, enforces zero trust for all forwarded traffic. Every allowed service must have an explicit rule before it. The simplicity is deceptive — the complexity is in the rules above it, but the principle is clear: deny everything, allow by exception.

Ubiquiti offers IDS/IPS as a built-in feature, which MikroTik does not have natively. The UniFi IDS/IPS inspects traffic for known attack signatures and can automatically block or alert on suspicious activity. For a home network, this provides a layer of protection that a MikroTik firewall alone does not — unless you add Suricata or Snort as a separate IDS on a mirror port.

The tradeoff: UniFi's IDS/IPS is a black box. You cannot customize the signatures, you cannot tune the sensitivity beyond broad categories, and you cannot export the logs to a SIEM easily. MikroTik's firewall is transparent — every rule is visible, every log entry is configurable, and you can pipe logs to an external syslog server or Loki stack for analysis.

Winner: Tie. MikroTik wins for transparent, automatable, zero-trust firewall rules. UniFi wins for built-in IDS/IPS that requires zero configuration. The ideal setup uses both — MikroTik as the router/firewall, UniFi APs for wireless, and an external IDS like Suricata for deep packet inspection.

VPN

Both platforms support WireGuard and OpenVPN. The implementation differences matter.

MikroTik RB5009 runs WireGuard natively in RouterOS 7. The WireGuard implementation supports multiple peers, fine-grained allowed IPs, and persistent keepalive — everything you need for site-to-site or remote access VPN. I run WireGuard on my RB5009 for remote access to the homelab, and the Terraform code for it is in the repo:

resource "routeros_interface_wireguard" "wg0" {
  listen_port = 13231
  name        = "wg0"
}

MikroTik also supports IPsec (IKEv1 and IKEv2) for site-to-site VPN with other vendors, OpenVPN for legacy client access, and PPTP (which you should never use). The RB5009's CPU handles WireGuard encryption in software — it maxes out around 1.5-2Gbps throughput, which is more than enough for a home WAN link.

UniFi UDR supports WireGuard and OpenVPN through the controller interface. Setup is simpler — a few form fills and you have a VPN tunnel. The UDR also supports Teleport (Ubiquiti's proprietary VPN) which uses WireGuard under the hood but with a simplified peer exchange mechanism.

The limitation: UniFi's VPN integration is shallow. You cannot fine-tune WireGuard parameters (MTU, pre-shared keys, handshake intervals) the way you can on MikroTik. Site-to-site VPN between UniFi and non-UniFi devices is possible but not well-documented. For a homelab that needs VPN access to specific VLANs or subnets, MikroTik's granular control is essential.

Winner: MikroTik RB5009 for VPN flexibility and fine-grained control. UniFi wins for quick-and-easy remote access setup.

VLAN Support

Both platforms support VLAN tagging, trunk ports, and access ports. The implementation approach is fundamentally different.

MikroTik handles VLANs through Bridge VLAN Filtering on a single bridge. All physical ports join one bridge, and VLANs are defined as entries in the bridge's VLAN table. This approach enables hardware offloading on the RB5009's switch chip — tagged traffic is switched in hardware, not software. The result: full wire-speed VLAN performance even with dozens of VLANs configured.

The Terraform code for my entire VLAN matrix — five VLANs, trunk ports, access ports, IP addresses, and DHCP servers — is about 100 lines of HCL. Add a new VLAN? Change one map and run terraform apply.

UniFi handles VLANs by creating "VLAN Networks" in the controller. Each VLAN is a separate network object with its own subnet, DHCP scope, and firewall rules. Trunk ports are configured per-switch-port in the UI. The approach is intuitive — each VLAN is a first-class object in the UI — but it ties you to the UniFi Controller for management.

Both approaches work well. MikroTik's is more flexible and automatable. UniFi's is more visual and self-documenting. For a homelab that needs VLANs for network segmentation (management, server, DMZ, IoT, admin), both platforms handle it without issue.

Winner: Tie. MikroTik for automation and hardware offloading. UniFi for visual management and simplicity.

Terraform & Automation Support

This is where MikroTik pulls away decisively for any homelab running infrastructure-as-code.

The MikroTik Terraform Provider (routeros) is mature, well-maintained, and covers nearly every RouterOS resource: interfaces, bridges, VLANs, firewall rules, DHCP, DNS, WireGuard, routing, and more. You can define your entire network stack in HCL, version it in Git, and apply changes through a CI pipeline or Atlantis. This is exactly how I manage my homelab — every MikroTik configuration change goes through a PR, gets reviewed, and is applied via Atlantis.

The Ubiquiti Terraform Provider (ubiquiti) is unofficial and limited. The community provider (paultyng/ubiquiti) covers basic device management but does not support network configuration, VLAN creation, or firewall rules through Terraform. There is no official UniFi Terraform provider from Ubiquiti. This means every network configuration change on UniFi must go through the controller UI — there is no IaC path.

For a homelab that already uses Terraform for Proxmox VMs, MikroTik firewall rules, and Kubernetes resources, the lack of UniFi Terraform support is a dealbreaker. You would end up with two configuration workflows: Terraform for everything except the network, and manual UI clicks for the network. That is exactly the kind of configuration drift that IaC is designed to prevent.

Winner: MikroTik RB5009 — decisively. No meaningful competition in this category.

Ecosystem & Expandability

Ubiquiti wins on ecosystem breadth. The UniFi ecosystem includes switches, access points, cameras (Protect), phones (Talk), displays (Connect), and a centralized controller that manages all of it. If you want one vendor for your entire network stack — router, switches, APs, cameras — UniFi provides that experience. The ecosystem lock-in is real, but so is the integration quality.

MikroTik has a broader product line than most people realize — routers, switches, access points, antennas, and even outdoor wireless gear. CAPsMAN (Centralized Access Point Manager) lets you manage MikroTik APs from the router itself. But the ecosystem is not as tightly integrated as UniFi. Each device runs RouterOS independently; there is no single controller dashboard (The Dude exists but is not comparable to the UniFi Controller).

For a homelab that primarily needs a router and a few switches, MikroTik's ecosystem is sufficient. For a whole-home networking deployment with cameras, multiple APs, and managed switches, UniFi's ecosystem provides a more cohesive experience.

Winner: Ubiquiti UDR for integrated ecosystem. MikroTik for standalone router/switch deployments.

Price & Value

The RB5009 costs roughly €180. The UniFi Dream Router costs roughly €300. That €120 difference buys you: built-in Wi-Fi 6 on the UDR (MikroTik requires a separate AP, ~€50-100 for a hAP ac3 or cAP ax), the UniFi Controller software, and IDS/IPS.

If you add a MikroTik AP to the RB5009 (which you will need if you want Wi-Fi), the total cost approaches the UDR. But you get a significantly more powerful router and a separate, dedicated AP — which is arguably better than a combined router/AP unit that must serve both roles.

For a homelab that does not need Wi-Fi from the router (most homelabs have a separate AP or run wired-only), the RB5009 at €180 is the clear value winner. For a whole-home setup where one device handles routing and Wi-Fi, the UDR's combined value is harder to beat.

Winner: MikroTik RB5009 for homelab use. UDR for whole-home Wi-Fi + routing in one device.

Who Should Buy Which

Buy the MikroTik RB5009 if:

You run Terraform for infrastructure management and want network automation
You want zero-trust firewall rules with full control over every packet
You value learning networking over having it pre-configured
You do not need Wi-Fi from the router itself
Budget matters — €180 buys enterprise-grade routing

Buy the UniFi Dream Router if:

You want plug-and-play setup with a polished UI
You need built-in Wi-Fi 6 without a separate AP
You want IDS/IPS without deploying a separate appliance
You are building a whole-home UniFi ecosystem (switches, APs, cameras)
You do not want to learn CLI-based network configuration

Buy both (MikroTik router + UniFi AP) if:

You want the best of both worlds: MikroTik routing/firewall automation with UniFi wireless management
This is what I would do if I were starting a new homelab today. MikroTik for the router, UniFi for the APs. The MikroTik handles VLANs, firewall, and VPN via Terraform. The UniFi AP handles wireless through its controller. Two vendors, two strengths, zero compromise.

Frequently Asked Questions

Can MikroTik and UniFi devices coexist on the same network?

Yes, they work together without issue. MikroTik handles routing and VLAN segmentation; UniFi APs broadcast the VLAN-tagged SSIDs. The key configuration point: the switch port connecting the UniFi AP must be a trunk port carrying all VLAN SSIDs. This is configured on the MikroTik side (bridge VLAN filtering) and the UniFi side (port profile with VLAN tagging). Both platforms handle this standard configuration well.

Is MikroTik really harder to set up than Ubiquiti?

Yes, initially. MikroTik's RouterOS has a learning curve that takes 2-4 weeks to climb for basic proficiency. Ubiquiti's UniFi can be configured in an afternoon. But the long-term payoff is significant: once you understand RouterOS, you can configure any networking scenario without vendor lock-in. The knowledge transfers to Cisco, Juniper, and every other networking platform. UniFi knowledge stays within the UniFi ecosystem.

Should I use MikroTik for my home Wi-Fi?

Not directly. MikroTik makes access points (hAP ac3, cAP ax), but their wireless performance and roaming are inferior to UniFi's. MikroTik excels at routing and switching; let a dedicated AP handle wireless. Pair an RB5009 with a UniFi U6 Lite or U6 Pro for the best of both worlds.

How does MikroTik security compare to Ubiquiti?

MikroTik's firewall is more configurable but requires manual hardening. RouterOS 7 has had security vulnerabilities (WinBox CVE in 2018, Chimay Red attacks), but MikroTik patches quickly and the attack surface is manageable with proper firewall rules and by disabling unused services. UniFi's IDS/IPS provides automated threat detection that MikroTik lacks natively, but the black-box nature of the detection limits its utility for security-conscious users who want full visibility.

Can I manage MikroTik with Terraform?

Yes, the routeros Terraform provider is mature and actively maintained. It covers interfaces, bridges, VLANs, firewall rules, DHCP, DNS, WireGuard, routing, and more. I manage my entire MikroTik RB5009 configuration through Terraform — every change goes through Git, gets reviewed, and is applied via Atlantis. The provider is at version 1.x and supports RouterOS 7.x. See the Terraform VLAN filtering article for a real-world example.

What is the best MikroTik router for a home lab?

The RB5009 is the sweet spot. It has 7 Gigabit ports (enough for most homelabs), a 10G SFP+ uplink for future-proofing, a switch chip for hardware-offloaded VLANs, and RouterOS 7 with full feature support. The hEX S (€70) is the budget option but lacks the 10G port and has a less capable switch chip. The CCR2004 (€300+) is overkill for home use unless you need 10Gbps routing throughput.

Conclusion

This is not a close comparison — but it is not a one-sided one either.

MikroTik wins for any homelab running infrastructure-as-code. The Terraform integration, the granular firewall control, the hardware-offloaded VLAN switching, and the 10G SFP+ uplink make the RB5009 the definitive choice for engineers who want to own their network stack. At €180, it is also the better value for routing performance.

Ubiquiti wins for whole-home networking where ease of use, ecosystem integration, and built-in Wi-Fi matter more than IaC automation. The UniFi Controller is the best consumer networking management platform available, and the ecosystem of switches, APs, and cameras is genuinely compelling.

For a homelab running Proxmox, K3s, and Terraform: buy the MikroTik RB5009. The learning curve pays for itself in flexibility, control, and the ability to manage your network the same way you manage your servers — as code.

🚀 Azure Zero-Trust Starter Pack — €99

Includes: Acmebot Enterprise VNet, Hub & Spoke Zero-Trust, Azure Firewall Forced Tunneling
Save €48 vs. buying the three modules separately
Bonus: ISO 27001 Auditor Checklist (PDF) - Annex A control map for all 3 modules
Everything to pass your first Azure compliance audit
Full source code for all 3 modules - no lock-in, no black box

Get the enterprise module →

Full source code · one-time payment · instant download

DEV Community: david

Vault Auto-Unseal Without Cloud KMS: The Polling Sidecar Pattern

The Sidecar

The Security Trade-off

Why 5 Seconds

The ExternalSecret Dependency Chain

The Network Policy

What I'd Change

Chaos Mesh in a Homelab: Weekly Pod-Kills on a Single-Host Cluster

The Setup

What Broke: First Week

What Broke: Network Latency

What It Taught

1. PDBs aren't optional

2. Init containers are single points of failure

3. Latency exposes timeout assumptions

4. Chaos testing on a homelab isn't overkill

Cloudflare Tunnel Without Opening a Single Firewall Port

The Architecture

Split-DNS on AdGuard

The Chunked Encoding Fix

Atlantis Through Traefik

DNS Record Management

K3s on Raspberry Pi: Why I Said No (And You Probably Should Too)

The Attempt

Why It Failed: SD Card Fragility

Why It Failed: Network Latency

Why It Failed: Resource Contention

The Deliberate Reversion

What the Pis Do Instead

The General Lesson

Azure Logic App Standard: Private Storage Needs Four Private Endpoints

The Symptom

The Root Cause

Why Four Plausible Fixes Failed

The Fix

The Generic Rebuild

The Checklist

The Takeaway

Velero Said Backups Succeeded. The Data Was Never There.

What Velero Actually Backs Up

The Discovery

The Fix

Verification

The Broader Problem: Backup Circularity

What Would Have Caught This

Self-Hosted SSO for 25 Services: Authelia OIDC on Kubernetes

The Architecture

The OIDC Configuration

Secrets in Vault

The Postgres Backend

The Authelia Schema Bug

Access Control Rules

The Two-Replica Setup

What I'd Change

My Terraform Runner Destroyed Itself Mid-Apply

The Setup

The Kill Shot

Why terraform plan Doesn't Catch This

The Fix: Move Atlantis Off k3s

The Ansible Layer

The Pattern

The ZFS ARC Freeze: How a Marginal PSU Killed My Entire Homelab

The Symptoms

The PSU Theory

The ZFS Deadlock

The Fixes

Fix 1: Disk I/O Isolation

Fix 2: ZFS ARC Limits

Fix 3: txg Timeout Tuning

Fix 4: Staggered Boot Order

What It Taught Me

How a Single Volume Was 65% of My Velero Backup and What I Almost Excluded Instead

The Symptom

The Investigation

The First Hypothesis (Wrong)

The Right Fix

The Exclusion

Result

The Pattern

Why `terraform plan` Doesn't Catch This