DEV Community: david

NFS vs local-path: The SQLite Trap That Corrupted My S3 Metadata

david — Mon, 14 Sep 2026 12:43:37 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Garage S3, my self-hosted S3-compatible object store, uses SQLite for its metadata database. One morning, Terraform state operations started failing with SQLITE_CORRUPT. The bucket metadata was gone. The terraform-state bucket, the Atlantis lock table, the Velero backup index — all of it stored in a SQLite file that was now corrupted.

The root cause: Garage was running on an NFS-backed PersistentVolume. NFS doesn't support the file-locking primitives that SQLite's WAL (Write-Ahead Logging) mode requires. Under concurrent access — Garage's metadata writer and a Velero backup reading the same database — the NFS lock delegation failed silently, and SQLite wrote to overlapping pages.

View the complete homelab infrastructure source on GitHub 🐙

The Storage Classes

My k3s cluster has two StorageClasses:

# NFS — for most workloads
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-client
provisioner: nfs-subdir-external-provisioner
parameters:
  server: 10.0.20.100
  path: /archive
reclaimPolicy: Retain

# Local-path — for embedded databases
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-path
provisioner: rancher.io/local-path
reclaimPolicy: Retain

NFS (nfs-client) is the default. It's backed by a dedicated LXC running an NFS server on ZFS, providing storage that survives pod rescheduling — a pod on k3s-12 can access the same PVC as a pod on k3s-13 because the NFS server is independent of any specific node.

Local-path (local-path) pins the PV to whichever node created it. If the pod reschedules to a different node, the PVC is inaccessible until the pod returns to the original node. This is a limitation, but it's the right trade-off for certain workloads.

Why SQLite and NFS Don't Work

SQLite's WAL mode requires fcntl() file locks — specifically, F_SETLK (non-blocking lock) and F_SETLKW (blocking lock). These locks coordinate access between concurrent processes writing to the same database file.

NFS handles file locks differently:

NFSv3: No native lock support. fcntl() calls return success but locks are local to the client — two NFS clients can both acquire an "exclusive" lock on the same file simultaneously.
NFSv4: Has LOCK operations, but the lock delegation model introduces latency and failure modes that SQLite's tight locking loop doesn't tolerate. If the NFS server is slow to respond to a lock request, SQLite's default 5-second busy timeout can expire, causing the application to retry — and the retry can conflict with the lock held by another client.
Kubernetes NFS provisioner: The nfs-subdir-external-provisioner uses NFSv4, but the lock delegation is handled by the NFS server's rpc.lockd daemon, which runs in a separate process space. Under concurrent load, lockd can lose track of which client holds which lock.

The result: SQLite thinks it has an exclusive lock, but another process (or the same process on a different connection) also has a lock. Both write to the database file. Pages overlap. The database corrupts.

The Garage Incident

Garage runs with two storage mounts:

# kubernetes/apps/garage/garage.yml
volumes:
  - name: data
    persistentVolumeClaim:
      claimName: garage-data      # NFS — bucket objects
  - name: meta
    persistentVolumeClaim:
      claimName: garage-meta      # local-path — SQLite metadata

The data volume (bucket objects) is on NFS — fine, because S3 object storage doesn't use file locks for individual files. The meta volume (SQLite database) was also on NFS initially. This worked until a Velero backup and a Terraform state write happened simultaneously.

Velero reads Garage's S3 API to enumerate backup objects. Terraform reads the same database to verify state file existence. Both hit SQLite through Garage's metadata layer. Under NFS, the concurrent reads triggered the lock delegation failure, and SQLite corrupted pages 169–184 of db.sqlite.

The Recovery

SQLite has a .recover command that can extract data from a corrupted database:

# Dump recoverable data from corrupted SQLite
sqlite3 db.sqlite ".recover" > recovered.sql

# Recreate the database from the dump
sqlite3 db_clean.sqlite < recovered.sql

The .recover command scanned every page of the corrupted file and extracted whatever data it could read. For pages 169–184 (the corrupted range), it found partial data — enough to reconstruct the bucket and key metadata, but not enough to guarantee referential integrity.

After recovery, the missing objects (terraform-state bucket and Atlantis lock key) had to be re-inserted manually via Python:

import sqlite3
import msgpack

conn = sqlite3.connect("db_clean.sqlite")
# Garage uses msgpack-encoded metadata with 'G2key'/'G2bkt' prefixes
# Re-insert the terraform-state bucket
conn.execute("INSERT INTO buckets (name, ...) VALUES (...)")
conn.commit()

The data was restored, but the trust was gone. The database could corrupt again under the same conditions.

The Fix

Move every embedded database to local-path:

# Garage — meta volume moved to local-path
volumes:
  - name: meta
    persistentVolumeClaim:
      claimName: garage-meta      # NOW: local-path (was: nfs-client)

The apps that need local-path:

App	Database	Why local-path
Garage S3	SQLite (metadata)	File-locking requirements
Mealie	SQLite (recipes)	WAL mode + concurrent access
Home Assistant	SQLite (state)	Inotify-based DB writes
Authelia	PostgreSQL (CNPG)	CNPG manages its own PV

PostgreSQL (via CNPG) doesn't have the SQLite lock problem because it uses its own file locking, but CNPG requires local-path or a CSI driver that supports ReadWriteOnce — NFS's ReadWriteMany semantics can confuse CNPG's WAL archiving.

The Trade-off

local-path means the PVC is pinned to one node. If the pod reschedules, it loses access to the data. For Garage, this is acceptable: Garage runs on a single node, and if that node goes down, the S3 data is unavailable regardless (it's on the same host).

For databases that need HA (Postgres, Redis), the solution isn't local-path or NFS — it's a managed operator (CNPG for Postgres) that handles replication and failover independently of the storage layer.

The principle: if the application uses file-level locking (SQLite, BoltDB, LMDB), it goes on local-path. If it uses network-level locking (PostgreSQL, MySQL), it goes on NFS or a managed operator.

SQLite on NFS is the same failure mode as running SQLite on an SMB share in a Windows domain: the file-locking semantics are fundamentally incompatible. In Azure, this maps to Azure Files (SMB-backed) vs. Azure Disk (block storage). Azure Files supports SMB locks but has the same delegation latency issues under concurrent access — any application that needs tight file-level locking should use Azure Disks, not Azure Files. The principle is identical: embedded databases need local, low-latency storage with native file-locking support.

Designing Data-Intensive Applications* covers exactly this class of correctness assumption — what a storage layer actually guarantees about concurrent access versus what an application silently assumes it guarantees — in far more depth than a corrupted db.sqlite file teaches you in the moment.

My Media Stack Lives in Two Containers and a Python CronJob

david — Mon, 14 Sep 2026 12:37:38 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

My media acquisition stack — SABnzbd for Usenet downloads, Sonarr for TV, Radarr for movies, Bazarr for subtitles, NZBHydra2 for indexer search — used to run as k3s Deployments. It worked, but three problems made it a bad fit for Kubernetes: GPU passthrough for Jellyfin transcoding, per-flow traffic isolation for indexer queries versus the actual download path, and the NFS file-locking trap for media libraries.

The solution: move the media stack out of k3s entirely. Jellyfin runs in its own GPU-passthrough LXC. The acquisition stack runs in a second LXC with Docker Compose. A Python CronJob in k3s bridges the two via Traefik Service+Endpoints.

Update: the first version of this stack wrapped the whole acquisition LXC in a Mullvad WireGuard tunnel via gluetun, on the assumption that every flow out of that box needed VPN protection equally. Two days later I tore that back out — see "The Traffic-Isolation Rethink" below for why a single blanket tunnel was the wrong model for what these five apps actually do.

View the complete homelab infrastructure source on GitHub 🐙

Why Kubernetes Was Wrong for Media

GPU Passthrough

Jellyfin needs GPU access for hardware video transcoding. The BMAX Mini PC* has an AMD Radeon Vega iGPU that supports VAAPI hardware transcoding. Proxmox GPU passthrough requires IOMMU group isolation — the GPU is passed to a single container or VM exclusively.

Kubernetes doesn't natively support GPU passthrough for LXCs. The nvidia-device-plugin works for NVIDIA GPUs on specific cloud providers, but for AMD iGPU passthrough on bare-metal Proxmox, you need a dedicated LXC with /dev/dri/renderD128 mapped directly.

Jellyfin runs in ct-srv-jellyfin-01 with:

# terraform/stacks/proxmox/lxc.tf
lxc_conf {
  desc = "Jellyfin - GPU passthrough"
  # GPU device mapped via pct set
}

k3s's own nodes here (vm-srv-k3s-11/12/13) are Proxmox VMs, not LXCs — a VM doesn't share the host kernel, so there's no "just bind-mount the device node" path the way there is for an LXC. Getting a k3s pod real GPU access on this hardware would mean classic VFIO passthrough of the iGPU to one specific VM: unbinding amdgpu from the Proxmox host and handing the whole device to vfio-pci instead. I looked at this seriously before ruling it out, because "GPU-in-Kubernetes" device plugins exist and I wanted to know if they'd apply.

They don't, for this hardware. AMD's own rocm/k8s-device-plugin targets ROCm compute (HIP/OpenCL) — a materially heavier stack than what VAAPI hardware transcoding actually needs, which is just /dev/dri visibility. And a Kubernetes device plugin can't manufacture GPU access a node's kernel doesn't already have — for a VM, that access only exists after real hypervisor-level VFIO passthrough, which a plugin doesn't do. Worse, this is a single consumer Ryzen APU, not a data-center part with SR-IOV or mediated-device support for splitting one GPU across VMs — passthrough would hand the entire GPU to exactly one of the three k3s VMs, and the Proxmox host itself (which uses amdgpu for its own display/telemetry) would permanently lose access to it. There's also no portability payoff to offset that cost: k3s's scheduler can't move a pod needing a passed-through device to a different node than the one VFIO was bound to, so the usual "GPU follows the pod" reason people move transcoding into Kubernetes never materializes on a single-host, single-iGPU homelab. It would just be Jellyfin running on a VM instead of an LXC, at the permanent cost of the GPU being unavailable to anything else on the host.

The LXC path avoids all of that: an LXC shares the host's kernel, so the host keeps the amdgpu driver bound and simply grants the container access to the resulting /dev/dri/renderD128 device node. Non-exclusive from the host's perspective, already proven working, no PCI device binding to get wrong. If this box ever gets a GPU with real SR-IOV support, this is worth revisiting — the constraint here is the specific hardware, not a principled objection to GPU workloads in Kubernetes.

The Traffic-Isolation Rethink

The acquisition LXC has two genuinely different outbound flows, and my first pass treated them as one problem. SABnzbd connects to Eweka (my Usenet provider) over NNTPS on port 563 — already encrypted end-to-end, and Usenet copyright enforcement works exclusively via BitTorrent peer-list monitoring, so there's no mechanism by which an ISP or rights-holder observes or reports Usenet downloads in the first place. NZBHydra2's indexer search queries are a completely different flow: plain HTTP/HTTPS lookups against third-party indexer sites, which does expose the home IP to whoever's on the other end, the same as browsing any site directly.

Wrapping the whole LXC in gluetun/Mullvad "solved" both at once, but it was the wrong tool for either: VPN on the download path halves throughput for no privacy benefit Eweka's own SSL doesn't already provide, and it risks Eweka flagging the account for apparent multi-subscriber IP sharing. I tore gluetun out two days after standing it up and replaced it with a model that actually matches the two flows:

SABnzbd → Eweka, direct, over SSL. No VPN. The ISP sees "connected to news.eweka.nl," never content — that's the whole job done by NNTPS alone.
NZBHydra2 → indexers, through a Tor SOCKS5 proxy, tor container at 172.28.1.10:9050, configured with no direct-connection fallback. Search queries are small and latency-tolerant, a good fit for Tor's limited bandwidth, and Tor is a better fit than a commercial VPN for metadata-only queries — no single operator to trust, no throughput to throttle. If Tor is down, indexer queries fail outright instead of silently leaking the home IP.

I verified this configuration directly rather than trusting the design intent on paper: sabnzbd.ini shows socks5_proxy_url = "" (empty — SABnzbd was never routed through Tor) with ssl = 1, ssl_verify = 2 confirming the direct-to-Eweka SSL path is real; nzbhydra.yml shows proxyType: SOCKS pointed at the Tor container with no fallback option enabled. One live exception I noticed and haven't chased down yet: one specific indexer bypasses Tor and connects directly (proxyIgnoreDomains) — possibly a site that blocks Tor exit nodes, possibly a leftover exception from before I understood this stack properly. Worth revisiting.

Tor's bandwidth genuinely can't handle bulk transfers, and routing downloads through it would be abusive to a network that exists for people who need anonymity for safety — never route the actual download path through Tor, only small metadata lookups.

The acquisition LXC (ct-srv-media-acq-01) runs all five apps this way — no blanket tunnel, no kill-switch sidecar to maintain, no shared failure mode between "is Eweka's SSL up" and "is the VPN provider's WireGuard endpoint reachable today."

NFS File Locking

Media libraries (downloaded files, metadata databases) live on NFS. As I wrote about in the SQLite trap article, NFS file-locking semantics don't work with embedded databases. Sonarr and Radarr use SQLite internally for their media databases — and those databases corrupt on NFS under concurrent access.

Moving the acquisition stack to a local LXC with local storage eliminates the NFS lock problem. The media files themselves (downloaded episodes, movies) still live on NFS for sharing, but the application databases stay local.

The Architecture

┌─────────────────────────────────────────────┐
│  k3s Cluster                                │
│  ┌─────────────────────────────────────┐    │
│  │ Python CronJob (every 10 min)       │    │
│  │ - Checks Sonarr/Radarr queue        │    │
│  │ - Clears stuck items                │    │
│  │ - Triggers Jellyfin library scan    │    │
│  └──────────┬──────────────────────────┘    │
│             │ HTTP via Traefik              │
│  ┌──────────▼──────────────────────────┐    │
│  │ Traefik IngressRoutes               │    │
│  │ - sabnzbd.woitzik.dev               │    │
│  │ - sonarr.woitzik.dev                │    │
│  │ - radarr.woitzik.dev                │    │
│  │ - bazarr.woitzik.dev                │    │
│  └─────────────────────────────────────┘    │
└──────────────────┬──────────────────────────┘
                   │ Traefik Service+Endpoints
┌──────────────────▼──────────────────────────┐
│  ct-srv-media-acq-01 (LXC, Tor for indexers)│
│  ┌─────────┐ ┌────────┐ ┌────────┐         │
│  │ SABnzbd │ │ Sonarr │ │ Radarr │         │
│  └─────────┘ └────────┘ └────────┘         │
│  ┌─────────┐ ┌──────────────┐              │
│  │ Bazarr  │ │ NZBHydra2    │              │
│  └─────────┘ └──────────────┘              │
└─────────────────────────────────────────────┘
┌─────────────────────────────────────────────┐
│  ct-srv-jellyfin-01 (LXC, GPU passthrough)  │
│  ┌─────────┐ ┌──────────────┐              │
│  │ Jellyfin│ │ /dev/dri/    │              │
│  │         │ │ renderD128   │              │
│  └─────────┘ └──────────────┘              │
└─────────────────────────────────────────────┘

The Traefik Bridge

k3s Traefik routes external traffic to services inside the cluster. But the media stack isn't in the cluster — it's in LXCs. The bridge: Traefik IngressRoutes point at Kubernetes Services, which use Endpoints objects with hardcoded IP addresses pointing at the LXC containers.

# kubernetes/apps/jellyfin/jellyfin.yml
apiVersion: v1
kind: Service
metadata:
  name: jellyfin
  namespace: apps
spec:
  ports:
    - port: 8096
      targetPort: 8096
---
apiVersion: v1
kind: Endpoints
metadata:
  name: jellyfin
  namespace: apps
subsets:
  - addresses:
      - ip: 10.0.20.254  # ct-srv-jellyfin-01
    ports:
      - port: 8096

This is the same pattern for all external services. The Service has no selector — it's a "headless" Service where the Endpoints are manually maintained. Traefik doesn't know or care that the backend is an LXC instead of a pod.

The IngressRoute for Jellyfin:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: jellyfin
  namespace: apps
spec:
  entryPoints: [websecure]
  routes:
    - match: Host(`media.woitzik.dev`)
      kind: Rule
      middlewares: [{name: authelia}]
      services:
        - name: jellyfin
          port: 8096
  tls:
    secretName: wildcard-woitzik-dev-tls

The media stack gets Authelia protection, wildcard TLS, and Cloudflare Tunnel external access — the same as every in-cluster service. The only difference is the backend IP.

The Python CronJob

A Python CronJob runs every 10 minutes inside k3s, bridging the gap between the media stack and the cluster:

# CronJob that monitors media acquisition
schedule: "*/10 * * * *"
jobTemplate:
  spec:
    template:
      spec:
        containers:
          - name: media-watchdog
            image: python:3.12-slim
            command:
              - python
              - /scripts/media-watchdog.py

The script:

Queries Sonarr/Radarr APIs for stuck queue items (downloads stuck for >30 minutes)
Clears stuck items and re-triggers the download
Checks Jellyfin's library scan status
Posts status to Discord via webhook

Without this CronJob, stuck downloads sit indefinitely. Sonarr and Radarr don't have built-in queue monitoring — they trust the download client to report status, and SABnzbd sometimes silently fails without notifying the *arr stack.

What I'd Change

Use a proper service mesh for cross-boundary traffic. The manual Endpoints pattern works but is fragile — if the LXC IP changes, the Endpoints must be updated manually. A DNS-based service discovery (Headscale DNS entries, for example) would be more resilient.
Move the CronJob to a native LXC cron. The Python script runs in a k3s Pod but talks to LXC-hosted services. It has no business being in the cluster. A systemd timer on the media acquisition LXC would be simpler.

The media stack's departure from Kubernetes is the same pattern as running stateful workloads outside AKS: GPU workloads go to dedicated VMs with GPU passthrough, traffic-sensitive workloads get network-level isolation instead of a sidecar, and file-locking workloads go to local SSDs. Kubernetes excels at stateless, horizontally-scalable workloads. Media transcoding and acquisition are neither — they're stateful, single-instance, and hardware-dependent. The right platform for them is the bare metal underneath, not the orchestration layer on top.

Private AKS on Azure: No Public API Server, No Public Node IPs, No Default Outbound

david — Mon, 07 Sep 2026 19:17:12 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Run az aks create with the defaults and you get a public API server FQDN, a Standard Load Balancer giving every node its own path to the internet, and — unless you go out of your way to stop it — an admin kubeconfig that works from anywhere with the right token. None of that is a bug. It's the fast path, and for a demo cluster it's fine.

For a cluster inside a governed Hub & Spoke, it's three separate audit findings. This is the Terraform to close all three at once: a private API server, forced-tunneled egress through a firewall you already control, and private-endpoint-only ACR and Key Vault.

Get the base template free on GitHub 🐙

Target Architecture

                         ┌─────────────────────┐
                         │   Azure Firewall     │  ← you supply the IP
                         │  (existing / NVA)     │
                         └──────────┬───────────┘
                                    │ UDR: 0.0.0.0/0
                         ┌──────────┴───────────┐
                         │   vnet-aks            │
                         │                       │
              ┌──────────┴──────────┐  ┌─────────┴──────────┐
              │  snet-aks-nodes     │  │ snet-private-       │
              │  (default-deny NSG) │  │ endpoints            │
              │                     │  │  - ACR (Premium)     │
              │  Private AKS        │  │  - Key Vault         │
              │  (no public FQDN)   │  │                      │
              └─────────────────────┘  └─────────────────────┘

Three engineering decisions drive this, same as the Hub & Spoke module this one is designed to sit behind:

No public control-plane endpoint — reachable only from inside the VNet, a peered network, or a VPN
Forced tunneling, not default outbound — every node packet leaves through a firewall you control
Private-endpoint-only dependencies — ACR and Key Vault never touch the public internet either

Step 1: The API Server Has No Public FQDN

resource "azurerm_kubernetes_cluster" "this" {
  # ...
  private_cluster_enabled             = true
  private_dns_zone_id                 = azurerm_private_dns_zone.aks.id
  private_cluster_public_fqdn_enabled = false

  network_profile {
    network_plugin = "azure"
    network_policy = "azure"
    outbound_type  = "userDefinedRouting"
  }
}

private_cluster_public_fqdn_enabled = false is the setting people miss. private_cluster_enabled = true alone still publishes a public FQDN that resolves to nothing reachable — a fingerprintable artifact for no benefit. Turning it off means az aks show doesn't leak a hostname at all.

The cluster needs its own Private DNS Zone for the API server (privatelink.<region>.azmk8s.io), linked to the VNet — same DINE-policy-safe lifecycle.ignore_changes pattern as the Hub & Spoke module's centralized zones, because Azure Policy tagging fights this resource too.

Step 2: `userDefinedRouting` — the Setting That Actually Matters

network_plugin = "azure" and network_policy = "azure" get most of the attention in AKS networking guides. The setting that actually determines whether traffic is inspectable is outbound_type.

Left at its default (loadBalancer), AKS provisions its own Standard Load Balancer outbound rule and every node gets a direct, un-inspected path to the internet — a private API server with fully public node egress is a common half-measure that looks locked down in the portal and isn't.

resource "azurerm_route_table" "aks_egress" {
  route {
    name                   = "default-via-firewall"
    address_prefix         = "0.0.0.0/0"
    next_hop_type          = "VirtualAppliance"
    next_hop_in_ip_address = var.firewall_private_ip
  }
}

resource "azurerm_subnet_route_table_association" "aks_nodes" {
  subnet_id      = azurerm_subnet.aks_nodes.id
  route_table_id = azurerm_route_table.aks_egress.id
}

outbound_type = "userDefinedRouting" tells AKS to trust this route table instead of provisioning its own path. The Route Table has to exist and be associated to the node subnet before the cluster is created — depends_on enforces the ordering, because AKS validates the UDR is actually in place at cluster-creation time and fails otherwise.

Step 3: ACR and Key Vault Without a Public Endpoint

Nodes still need to pull images and (often) read secrets. The naive fix — admin_enabled = true on the registry, a shared pull secret in a Kubernetes Secret — is a credential that outlives any single deployment and shows up in kubectl get secret -o yaml for anyone with namespace read access.

resource "azurerm_container_registry" "this" {
  sku                            = "Premium" # required for Private Link
  admin_enabled                  = false
  public_network_access_enabled  = false
}

resource "azurerm_role_assignment" "aks_acr_pull" {
  scope                            = azurerm_container_registry.this[0].id
  role_definition_name             = "AcrPull"
  principal_id                     = azurerm_kubernetes_cluster.this.kubelet_identity[0].object_id
  skip_service_principal_aad_check = true
}

The kubelet identity — the identity that actually pulls images, not the cluster's control-plane identity — gets AcrPull directly. No credential to rotate, nothing in a Secret, nothing that outlives the node it's bound to. Key Vault follows the identical shape: RBAC-authorized, public_network_access_enabled = false, kubelet identity gets Key Vault Secrets User, ready for the CSI Secrets Store driver to mount without ever touching a static key.

What This Doesn't Do

It's a starting cluster, not a finished platform. No Entra ID–integrated kubectl auth (add azure_active_directory_role_based_access_control yourself), no multi-pool topology, no cluster autoscaler wiring, sku_tier defaults to Free (no uptime SLA — set Standard for production). And it assumes you already have a firewall to hand it a private IP — pair it with the Azure Firewall Forced Tunneling module if you don't.

For the operational reality of running production Kubernetes past the "it deployed" stage — the workflows and failure modes that don't show up in a getting-started guide — GitOps and Kubernetes* is the reference I've actually kept open while doing this.

Staggered VM Boot: How I Prevented a Load Average of 147

david — Mon, 07 Sep 2026 12:29:57 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

After a host reboot, every VM and LXC on the Proxmox host started simultaneously. Twelve containers and VMs, all booting at once, all hitting the same NVMe for root filesystem reads, service starts, and NFS mounts. The host load average hit 147.

For context: load average represents the number of processes in the run queue or waiting for I/O. On a 16-thread CPU, a load average of 147 means 147 processes are competing for CPU or disk time. Every service was slow to start, k3s took minutes to become ready, and DNS didn't resolve for the first 90 seconds because the Raspberry Pi DNS nodes were waiting for services that hadn't booted yet.

The fix wasn't more resources — it was boot order.

View the complete homelab infrastructure source on GitHub 🐙

The Boot Storm

When the Proxmox host starts, all VMs and LXCs configured with onboot=1 start simultaneously. The host's NVMe handles root filesystem reads for every container, plus the ZFS txg commits from the NFS server, plus etcd writes from the k3s control-plane.

The simultaneous startup creates a thundering herd: 12 processes all requesting I/O at the same time, the NVMe queue depth maxes out, I/O latency spikes, and services that depend on each other (k3s needs NFS, k3s apps need DNS, DNS needs k3s services) enter a cascading wait state.

The load average doesn't just spike and recover — it compounds. Services that fail to start within their timeout window retry, adding more processes to the queue. k3s control-plane tries to mount NFS volumes, NFS is slow because it's competing with 11 other containers for I/O, k3s retries the mount, adding more load.

The Fix: Staggered Boot Order

Proxmox supports startup order with delays. The configuration in Terraform:

# terraform/stacks/proxmox/lxc.tf

# NFS first — k3s depends on it
resource "proxmox_virtual_machine" "ct_srv_nfs_01" {
  # ...
  startup = {
    order = 1
    up    = 30  # wait 30s after boot before starting next
  }
}

# k3s control-plane — waits for NFS
resource "proxmox_virtual_machine" "vm_srv_k3s_11" {
  # ...
  startup = {
    order = 2
    up    = 30  # 30s after NFS is up
  }
}

# k3s workers — 30s apart from each other
resource "proxmox_virtual_machine" "vm_srv_k3s_12" {
  startup = {
    order = 3
    up    = 30
  }
}

resource "proxmox_virtual_machine" "vm_srv_k3s_13" {
  startup = {
    order = 4
    up    = 30
  }
}

The order:

NFS (order 1) — boots first, 30s head start. k3s PVCs mount from NFS, so NFS must be ready before k3s starts.
k3s-11 (order 2) — control-plane + etcd. Boots 30s after NFS. Needs NFS for system PVCs.
k3s-12 (order 3) — worker. Boots 30s after control-plane. Needs API server ready.
k3s-13 (order 4) — worker. Boots 30s after k3s-12.
LXCs (order 5+) — everything else. Docker workloads, media stack, DMZ.

Total boot sequence: ~3 minutes for everything to be online. Previously: all at once, 147 load average, 5+ minutes to stability.

Why NFS First

NFS is the foundation of the storage layer. Every k3s PVC (Authelia, Vaultwarden, Paperless, Nextcloud) mounts from the NFS server at 10.0.20.100. If NFS isn't ready when k3s starts, the pod mount attempts fail, Kubernetes retries with exponential backoff, and the pods sit in ContainerCreating for minutes.

NFS itself depends on ZFS — the NFS export directory lives on the ZFS pool. ZFS needs a few seconds after boot to complete any pending txg commits and mount the pool. The 30-second head start gives ZFS and NFS time to stabilize before k3s starts hammering them with mount requests.

The CPU Scheduling Priority

Beyond boot order, k3s VMs get CPU scheduling priority via cpu.units:

# k3s VMs get 2x scheduling priority
cpu {
  cores = 4
  units = 2048  # default is 1024
}

# LXCs stay at default priority
cpu {
  cores = 2
  units = 1024  # default
}

cpu.units tells the Proxmox scheduler how to weight CPU time when multiple VMs compete for the same physical cores. units=2048 means k3s VMs get twice the CPU scheduling priority over LXCs. When Ollama is running LLM inference on the AI LXC and k3s needs CPU for etcd fdatasync, etcd wins.

This matters during boot too: even with staggered starts, there's overlap between late-booting LXCs and already-running k3s workloads. The CPU priority ensures k3s gets scheduling preference.

The `onboot` Gotcha

Proxmox's bpg/proxmox Terraform provider doesn't reliably manage the onboot attribute. terraform plan always shows "No changes" regardless of the live value — a known limitation of the provider.

This means onboot must be set manually after any LXC recreate:

pct set <id> -onboot 1

I discovered this the hard way: after recreating a container via Terraform, onboot defaulted to 0. The next host reboot silently skipped that container. The k3s control-plane node came up without its NFS mount, and half the cluster was in CrashLoopBackOff until I noticed.

The workaround: a manual pct set step in the operations runbook, applied after every LXC creation. Not ideal, but documented and repeatable.

Before vs. After

Metric	Before (simultaneous)	After (staggered)
Peak load average	147	12
Time to k3s ready	5+ minutes	90 seconds
Time to DNS functional	90 seconds	30 seconds
I/O wait %	85%	15%
Failed mount attempts	12-15	0

The staggering didn't add total boot time — it redistributed the I/O load over 3 minutes instead of concentrating it in 30 seconds. Services come up later individually but the cluster as a whole is stable sooner because nothing is fighting for I/O. The same host also runs a memory overcommit guard for the RAM side of this problem — boot order fixes the I/O storm, the guard fixes the memory one.

Boot storm mitigation is the same problem in Azure: when you scale out a VMSS from 0 to 50 instances, all 50 hit the Azure fabric simultaneously. Azure handles this with staggered placement and shared disks, but the principle is identical — spread the I/O load over time instead of concentrating it. In a homelab, you do it yourself with startup.order and startup.up delays.

Designing Data-Intensive Applications* has a genuinely useful framing for this kind of thundering-herd problem, even though it's written about databases rather than hypervisors - the queueing math behind "everything wants the same resource at the same instant" doesn't care what the resource actually is.

The Overcommit Guard: How a Python Script Prevents Host Freezes

david — Thu, 03 Sep 2026 19:30:32 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

My Proxmox host froze three times in one week. The root cause was RAM pressure pushing ZFS into an I/O stall on a single shared NVMe. After fixing the freeze (ZFS tuning, cache=none+aio=native), I needed a way to prevent it from ever happening again — a CI guard that catches memory overcommit before it hits the host. It's the same host where uncontrolled boot storms once pushed load average to 147; memory pressure and I/O pressure are two versions of the same single-host bottleneck.

The first version of the guard was wrong. It summed VM and LXC memory allocations together against one ceiling, and flagged 85 GB as "allocated" on a host with 62 GB of physical RAM. A live check showed only 41 GB actually in use — 21 GB available. The guard was overstating pressure by 40 GB because it treated two different memory physics as the same thing.

The fix was a two-tier model that understands the difference between a VM reservation and an LXC ceiling.

View the complete homelab infrastructure source on GitHub 🐙

The Two Memory Physics

VM `dedicated` — Real Reservation

When you set dedicated = 12288 (12 GB) on a Proxmox VM, QEMU pre-allocates that memory as a host process. The moment the VM starts, 12 GB of physical RAM is gone — reserved for the QEMU process, not available for anything else. The host's free -h reflects this immediately.

This is a hard reservation. If the sum of all VM dedicated values plus ZFS ARC plus host overhead exceeds physical RAM, the host is overcommitted. The kernel will start swapping, ZFS will stall waiting for I/O, and the freeze pattern repeats.

LXC `dedicated` — Soft Ceiling

When you set dedicated = 4096 (4 GB) on a Proxmox LXC, you're setting memory.max in the cgroup. This is a ceiling the kernel enforces only if the container actually tries to use that much. It reserves nothing on the host up front.

A container with dedicated = 4096 might be using 800 MB. The host sees 800 MB, not 4 GB. The remaining 3.2 GB is available for other workloads. This is why the original guard was wrong: summing LXC dedicated values counts memory that isn't actually consumed.

A live check on the host confirmed this:

# Check actual LXC memory usage vs configured limits
for ct in $(pct list | awk 'NR>1 {print $1}'); do
  limit=$(pct config $ct | grep "dedicated" | awk '{print $2}')
  actual=$(pct exec $ct -- cat /sys/fs/cgroup/memory.current 2>/dev/null)
  echo "CT $ct: limit=${limit}MB actual=$((actual/1024/1024))MB"
done
# → Most CTs using 20-40% of their configured limit

The Two-Tier Guard

Hard Gate (CI-Fail)

The hard gate catches real overcommit risk. It sums only VM dedicated values (real reservations) plus ZFS ARC max plus a fixed host reserve:

# Hard gate: VM reservations + ARC + host reserve
ZFS_ARC_MAX_GB = 4       # /sys/module/zfs/parameters/zfs_arc_max
HOST_RESERVE_GB = 6       # kernel + QEMU overhead
HARD_GATE_CEILING_GB = 44 # safe ceiling for 62 GB physical

hard_gate_mb = vm_dedicated_total + (ZFS_ARC_MAX_GB * 1024) + (HOST_RESERVE_GB * 1024)

If the hard gate exceeds 44 GB, the CI build fails. The PR cannot be merged. This is intentional: a VM dedicated change that pushes past the ceiling is exactly the kind of change that caused the original freeze.

The ceiling (44 GB on a 62 GB host) leaves 18 GB of headroom for:

LXC actual usage (typically 8-12 GB across all containers)
Kernel and system overhead not captured in the reserve
Burst spikes from Ollama inference or Paperless OCR

Soft Check (Warn-Only)

The soft check sums LXC dedicated values and compares against physical RAM. If it exceeds 62 GB, a warning is printed — but the build does not fail. LXC ceilings are soft; summing them overstates real pressure.

# Soft check: LXC CT limits vs physical RAM (visibility only)
if lxc_dedicated_total > PHYSICAL_RAM_GB * 1024:
    print(f"WARN: LXC CT limits sum to {lxc_dedicated_total / 1024:.1f} GB, "
          f"over physical RAM ({PHYSICAL_RAM_GB} GB). "
          f"Check actual usage: pct exec <id> -- cat /sys/fs/cgroup/memory.current")

The soft check exists for visibility. If someone adds a new LXC with 32 GB dedicated and the sum crosses 100 GB, the warning fires — but it doesn't block the merge, because the actual usage is probably a fraction of the configured limit.

Why `floating` Is Ignored

VMs can have both dedicated (ceiling) and floating (balloon-driven floor). Under host memory pressure, Proxmox can deflate the balloon down to the floating minimum, freeing memory for other workloads.

The guard conservatively uses dedicated, not floating, because:

The balloon only deflates after pressure is detected — it doesn't prevent the pressure
A sudden allocation spike (Ollama loading a 26B model) can't wait for the balloon to deflate
The guard exists to catch pending risk, not to model steady-state usage

floating values are parsed and printed for visibility, never counted in the hard gate.

The CI Integration

The script runs in pre-commit hooks and GitHub Actions CI:

# .github/workflows/ci.yml
- name: Memory overcommit guard
  run: python scripts/check-host-memory-overcommit.py

A Terraform change that adds a new VM or increases a VM's dedicated value is checked against the ceiling before merge. If it pushes past 44 GB, the PR is blocked with a clear error message explaining why.

$ python scripts/check-host-memory-overcommit.py
REL-035 memory overcommit guard (two-tier model)

-- Hard gate (CI-fail): VM reservations + ARC + host reserve --
  VM `dedicated` sum (3 VMs): 36864 MB (36.0 GB)
  VM `floating` sum  (3 VMs): 40960 MB (40.0 GB) -- informational only, NOT counted
  + ZFS ARC max: 4 GB
  + host/hypervisor reserve: 6 GB
  = hard gate total: 47104 MB (46.0 GB)
  Ceiling: 44 GB

FAIL: hard gate total (46.0 GB) exceeds the 44 GB ceiling by 2.0 GB.

What It Prevents

The guard exists because mini — the single Proxmox host running this entire homelab — has no failover. A host freeze means every service is down simultaneously: k3s, databases, DNS, monitoring, backups. The only recovery is a hard power cycle.

REL-016 (the ZFS freeze) happened because RAM pressure pushed ZFS into a stall-wait state on the shared NVMe. The hard gate catches the most common path to that state: a VM dedicated change that leaves insufficient headroom for the kernel, ZFS, and LXC workloads.

It doesn't prevent every possible freeze — a runaway process inside a VM can still consume all its allocated memory and cause host pressure. But it prevents the planned overcommit: the Terraform change that accidentally pushes past the ceiling because someone added a new VM without checking the math. CPU has the same two-tier problem; see how cpu.units scheduling priority solved it for etcd.

Memory overcommit modeling is the same problem in Azure: Reserved VM instances guarantee physical memory allocation, while Burstable VMs share host memory and can be throttled. Mixing both in the same Availability Set without understanding the difference produces the same false-sense-of-security that my original guard had. The fix is the same: separate hard reservations from soft ceilings, and never sum them together.

GitOps for Firewall Rules: MikroTik + Terraform + Atlantis

david — Mon, 31 Aug 2026 13:31:54 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

My MikroTik RB5009 has no manual RouterOS configuration. Every firewall rule, every VLAN interface, every DHCP static lease, every NAT port-forward — all of it lives in Terraform, reviewed via Atlantis pull requests, and applied through the same GitOps workflow that manages the Kubernetes cluster. This is the delivery pipeline underneath the zero-trust default-deny firewall policy and the VLAN matrix — this article covers the Atlantis plumbing, not the rules themselves.

This article covers the full implementation: the locals block that drives the VLAN matrix, the place_before pattern for deterministic firewall ordering, and how a router's LED schedule ended up in Terraform.

View the complete homelab infrastructure source on GitHub 🐙

The Provider Setup

The routeros provider (v1.80+) connects to the MikroTik via the REST API:

# terraform/stacks/network/providers.tf
terraform {
  required_version = ">= 1.5"
  required_providers {
    routeros = {
      source  = "terraform-routeros/routeros"
      version = "~> 1.80"
    }
  }
  backend "s3" {
    bucket = "terraform-state"
    key    = "network/terraform.tfstate"
    endpoints = {
      s3 = "https://s3.woitzik.dev"
    }
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    skip_requesting_account_id  = true
  }
}

The S3 backend is Garage, the self-hosted S3-compatible storage running inside k3s. The Terraform state for the network stack lives on the same cluster that depends on the network — another circular dependency that's acceptable because a full cluster failure means the network is the least of your problems.

The VLAN Matrix

The locals block is the single source of truth for the entire network:

# terraform/stacks/network/main.tf
locals {
  homelab_vlans = {
    "vlan20-srv"    = 20
    "vlan30-dmz"    = 30
    "vlan40-iot"    = 40
    "vlan100-admin" = 100
  }

  rpi_port_mapping = {
    "ether6" = 20  # RPi 4B #1 (Keepalived Node A)
    "ether7" = 20  # RPi 4B #2 (Keepalived Node B)
  }

  proxmox_port = "ether5"

  # Port mapping for all VLAN-tagged trunks
  port_mapping = {
    "ether1" = 1   # WAN (FritzBox)
    "ether2" = 20  # k3s-11
    "ether3" = 20  # k3s-12
    "ether4" = 20  # k3s-13
    "ether5" = 20  # Proxmox host
    "ether6" = 20  # RPi #1
    "ether7" = 20  # RPi #2
    "sfp-sfpplus1" = 30  # DMZ switch
  }
}

Adding a new VLAN means one entry in homelab_vlans. Terraform auto-generates:

Bridge VLAN entries
DHCP server per VLAN
Firewall rules per VLAN
VLAN interface on the bridge

Moving a device between VLANs means changing one number in port_mapping. The firewall rules, DHCP leases, and bridge entries update automatically.

Deterministic Firewall Ordering

MikroTik evaluates firewall rules in order and stops at the first match. Terraform manages rule ordering via place_before references:

# terraform/stacks/network/firewall_deterministic.tf

# Final drop rule — must be last in the forward chain
resource "routeros_ip_firewall_filter" "fwd_99_drop_all" {
  action       = "drop"
  chain        = "forward"
  place_before = null  # explicit: this is the last rule
  comment      = "99: Global - Final Drop (Zero Trust Policy)"
}

# Anti-spoofing — must come before any accept rules
resource "routeros_ip_firewall_filter" "fwd_00b_anti_spoof" {
  action       = "drop"
  chain        = "forward"
  src_address  = "10.0.0.0/8"
  in_interface_list = "WAN"
  place_before = routeros_ip_firewall_filter.fwd_01_established.id
  comment      = "00b: Anti-Spoof - Drop internal src from WAN"
}

# Established/related — allows return traffic
resource "routeros_ip_firewall_filter" "fwd_01_established" {
  action       = "accept"
  chain        = "forward"
  connection_state = ["established", "related"]
  place_before = routeros_ip_firewall_filter.fwd_99_drop_all.id
  comment      = "01: Allow Established/Related"
}

The place_before attribute creates explicit ordering dependencies. Rule 00b must come before rule 01, which must come before rule 99. Terraform resolves these dependencies during plan, so the apply order matches the intended evaluation order.

Without place_before, Terraform would apply rules in dependency-graph order, which is not guaranteed to match the intended firewall evaluation order. A rule that should be evaluated first might be created last, allowing a broader rule to match traffic before the narrower, more specific rule is evaluated.

The full ruleset in firewall_deterministic.tf has 22 rules with explicit ordering. The firewall_extra.tf file has 19 additional rules for VPN access tiers, monitoring, and application-specific port forwards — all with place_before references to maintain correct evaluation order.

The Atlantis Flow

Every change to the network stack goes through Atlantis:

Edit terraform/stacks/network/*.tf
Push to a feature branch
Open a PR against main
Atlantis comments with the terraform plan output
Review the plan
Comment atlantis apply
Atlantis applies the changes to the MikroTik

The atlantis.yaml at the repo root defines the project:

version: 3
projects:
  - name: network
    dir: terraform/stacks/network
    workspace: default
    autoplan:
      when_modified: ["*.tf", "*.tfvars"]
      enabled: true

The plan output shows exactly what will change:

# routeros_ip_firewall_filter.fwd_04a_monitoring will be updated in-place
~ resource "routeros_ip_firewall_filter" "fwd_04a_monitoring" {
      action      = "accept"
      chain       = "forward"
  ~   dst_port    = "9100" -> "9100,9090"
      src_address = "10.0.20.0/24"
      comment     = "04a: SRV - Prometheus scrape to MGMT"
  }

No surprise changes. Every rule modification is visible in the PR before it touches the live router.

What Goes in Terraform vs. What Doesn't

In Terraform:

All firewall filter rules (input + forward chains)
VLAN interfaces and bridge matrix
DHCP static leases
NAT port-forwards
QoS traffic shaping
SNMP community configuration
Power LED scheduling (yes, really)

Not in Terraform:

RouterOS user management (done once, rarely changes)
Certificate renewals (handled by the router's own scheduler)
Custom scripts for monitoring (run via SNMP, not Terraform)

The principle: if it affects traffic flow or network behavior, it's in Terraform. If it's one-time configuration or the router's own maintenance, it stays in RouterOS.

The LED Night-Mode

The MikroTik RB5009 has a power LED that's bright enough to be annoying in a dark room. RouterOS supports LED scheduling via the system scheduler. This ended up in Terraform:

resource "routeros_system_scheduler" "led_night_mode_off" {
  name     = "led-off"
  start_time = "23:00:00"
  interval  = "00:24:00"
  on_event = "/system led set led1 state=off"
}

resource "routeros_system_scheduler" "led_night_mode_on" {
  name     = "led-on"
  start_time = "07:00:00"
  interval  = "00:24:00"
  on_event = "/system led set led1 state=on"
}

Two scheduler entries. LED off at 23:00, on at 07:00. Managed via Terraform, reviewed via PR, applied via Atlantis. It's the most trivial thing in the entire network stack, and it's also the change that made my partner the happiest.

The Anti-Patterns

Two things I deliberately avoid:

Never create resources directly via RouterOS API. Every resource must be created through Terraform. If I need to debug a live issue and make a direct API change, I immediately add a moved {} block or import {} block to bring that resource under Terraform management. This prevents the "77 rules, Terraform knows about 22" problem I wrote about earlier.
Never terraform apply locally. All applies go through Atlantis PRs. This creates an audit trail, forces me to review every change, and prevents accidental modifications during debugging sessions.

Firewall-as-code via Terraform is the same pattern in Azure: NSG rules, Azure Firewall Policy rule collections, and Application Security Groups are all Terraform resources. The difference is that Azure's NSG API handles rule ordering automatically based on priority numbers, while MikroTik requires explicit place_before dependencies. Both approaches achieve the same goal: firewall rules that are reviewed, version-controlled, and auditable.

When Your LLM Hallucinated Your OCR

david — Thu, 27 Aug 2026 15:05:30 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

I discovered that paperless-gpt was generating plausible-looking but entirely fabricated German OCR text for scanned documents. The model returned confident, grammatically correct German sentences for pages that were actually invoices, receipts, and handwritten notes. The titles it assigned to these documents were similarly fictional.

The root cause wasn't a bug in paperless-gpt. It was a configuration error that sent page images to a text-only model that couldn't see images at all — and the model did exactly what language models do when given input they can't process: it made something up.

The second root cause was underneath: the Ollama iGPU backend that was supposed to accelerate inference was crashing 451 times per day from an unstable Vulkan/radv driver fallback.

View the complete homelab infrastructure source on GitHub 🐙

The Configuration Error

The paperless-gpt deployment had two model settings:

# kubernetes/apps/paperless/paperless-gpt.yml
env:
  - name: LLM_MODEL
    value: "qwen2.5:7b-instruct"        # text-only model for tagging/title generation
  - name: VISION_LLM_MODEL
    value: "qwen2.5-coder:7b"           # ← BUG: this is also text-only

VISION_LLM_MODEL should point at a model that can process images. qwen2.5-coder:7b is a text-only code completion model — it has no vision capability, no image encoder, no way to process pixel data.

paperless-gpt's OCR provider sends page images directly to the model specified by VISION_LLM_MODEL. There's no fallback mode, no capability check, no error when the model can't process images. The model receives the image (encoded as base64 in the prompt), can't interpret it, and returns its best guess based on context clues in the prompt text.

For a scanned German invoice, the model would generate something like:

Rechnung Nr. 2024-0847
Betrag: €1.247,83
Firma: Mustermann GmbH

Confident. Grammatically correct. Entirely fictional. The real document might be a parking ticket, a utility bill, or a handwritten note — the model couldn't tell because it couldn't see the image.

How Many Documents Were Affected

The pipeline processed documents for several weeks with this misconfiguration. Every image-based document — scanned PDFs, photographed receipts, screenshots — received fabricated OCR content. Text-based documents (digital PDFs with selectable text) were unaffected because paperless-gpt uses the embedded text layer, not the vision model, for those.

The affected documents had titles like:

Rechnung Nr. 2024-1847 — Firma Schmidt & Partner
GEPRÜFT — Handelsregister HRB 12345

The titles were completely made up, but they looked real enough that I didn't notice until I searched for a specific invoice and found a document titled with a company name I didn't recognize. Checking the actual scanned image confirmed: the document was a utility bill from a different provider entirely.

The Fix

# kubernetes/apps/paperless/paperless-gpt.yml
env:
  - name: LLM_MODEL
    value: "qwen2.5:7b-instruct"
  - name: VISION_LLM_MODEL
    value: "minicpm-v"                   # actual vision-capable model

minicpm-v is a small (4B parameter) vision-language model that can process images and return text. It's not as capable as larger models, but it can actually see the page content and generate accurate OCR text.

After switching, previously affected documents needed re-OCR. paperless-gpt doesn't re-process documents automatically — the fabricated content was already committed to the Paperless database. Manual re-processing was required for each affected document.

The Ollama iGPU Problem

The configuration error was discoverable earlier if the Ollama backend had been stable. But the Ollama instance was running on the AI LXC with the AMD Barcelo iGPU's Vulkan/radv fallback, and it was crashing constantly.

The AMD Ryzen 7 5825U's integrated GPU (gfx90c) has no official ROCm support. The community workaround uses HSA_OVERRIDE_GFX_VERSION=9.0.0 to spoof a supported GPU, combined with the Vulkan/radv driver for compute. This configuration was never stable on this chip.

The crash pattern:

vk::DeviceLostError: the GPU has been lost

451 crashes in a single day. Each crash terminates the Ollama inference process, which kills any in-flight LLM request. paperless-gpt, Open WebUI, and any other service using Ollama would see connection refused errors.

The iGPU crashes were the actual cause of paperless-gpt failures that I initially attributed to paperless-gpt itself. The vision model was working correctly — it was Ollama crashing before the model could finish processing.

The fix: switch Ollama to CPU-only mode and remove the OLLAMA_IGPU_ENABLE=1 flag entirely.

# Ansible role for Ollama
env:
  OLLAMA_HOST: "0.0.0.0:11434"
  # OLLAMA_IGPU_ENABLE removed — CPU-only for stability
  # HSA_OVERRIDE_GFX_VERSION removed — was never stable on gfx90c

CPU inference is slower (qwen2.5:7b takes ~30 seconds per inference vs ~8 seconds on the iGPU), but it doesn't crash. Stability over speed for a homelab document processing pipeline — the same trade-off that governs what gets to auto-update vs. what needs a human on this cluster: a slower, boring path beats a fast, unstable one.

The Lesson: Two Failure Modes Stacked

The incident had two independent root causes that interacted to make debugging harder:

Vision model pointing at text-only model — fabricated OCR content
Ollama iGPU crashing 451x/day — intermittent connection failures

If only (1) existed, I'd have noticed the fabricated content immediately. If only (2) existed, I'd have blamed Ollama instability. Both together meant that paperless-gpt failures looked like Ollama crashes (intermittent, random) rather than a systematic configuration error (consistent, every image document).

The debugging lesson: when a system fails in two different ways (sometimes fabricated content, sometimes connection refused), check for two separate root causes rather than assuming one explains both — the same discipline that mattered when three merged GitOps fixes turned out to have three unrelated failure modes.

AI model misconfiguration in production has the same pattern: a model that can't process the input type it's given will produce plausible-looking but wrong output, and there's no automatic error. Azure OpenAI's vision models require explicit multimodal input formatting — sending a raw image to a text-only deployment doesn't error, it just ignores the image and responds to any text in the prompt. The fix is the same: verify that your model's capabilities match your input types, and test with known inputs before trusting the output.

Vault Auto-Unseal Without Cloud KMS: The Polling Sidecar Pattern

david — Mon, 24 Aug 2026 07:34:21 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

HashiCorp Vault Enterprise has auto-unseal: a sealed Vault automatically unseals using a cloud KMS (AWS KMS, Azure Key Vault, GCP Cloud KMS). Vault OSS doesn't. Every time a Vault pod restarts — node failure, OOMKill, Kubernetes rescheduling — someone has to manually unseal it with vault operator unseal using the unseal keys.

On a homelab cluster where Vault backs ExternalSecrets for 25+ services, a sealed Vault means every ExternalSecret refresh fails. Authelia can't start (no hmac-secret), Open WebUI can't start (no WEBUI_SECRET_KEY), and half the cluster sits in init-container loops waiting for secrets that Vault can't provide.

The fix: a polling sidecar that auto-unseals Vault OSS without a KMS. The trade-off is a collapsed security boundary — but for a homelab, it's the right trade-off.

View the complete homelab infrastructure source on GitHub 🐙

The Sidecar

The vault-unseal Deployment runs as a separate pod in the vault namespace, polling every 5 seconds:

# kubernetes/system/vault/unseal.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault-unseal
  namespace: vault
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vault-unseal
  template:
    spec:
      serviceAccountName: vault-unseal
      containers:
        - name: vault-unseal
          image: hashicorp/vault:1.21.4
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                # Check if Vault is sealed
                SEALED=$(vault status -format=json | jq -r .sealed)
                if [ "$SEALED" = "true" ]; then
                  # Read unseal keys from the Kubernetes Secret
                  for i in 1 2 3; do
                    KEY=$(kubectl get secret vault-unseal-keys \
                      -n vault -o jsonpath="{.data.key$i}" | base64 -d)
                    vault operator unseal "$KEY"
                  done
                  echo "Vault unsealed at $(date)"
                fi
                sleep 5
              done
          env:
            - name: VAULT_ADDR
              value: "http://vault.vault.svc.cluster.local:8200"
          resources:
            requests:
              cpu: 10m
              memory: 32Mi
            limits:
              cpu: 50m
              memory: 64Mi

The pod needs kubectl access to read the unseal keys from a Kubernetes Secret. The vault-unseal ServiceAccount has a Role that grants get on the vault-unseal-keys Secret only.

The Security Trade-off

In a KMS-based auto-unseal setup, the unseal keys are never stored anywhere accessible to the cluster. The KMS holds the master key, and Vault uses it to decrypt the master key that seals the storage. An attacker who compromises the cluster can't unseal Vault because the KMS key is outside the cluster boundary.

In the polling sidecar pattern, the unseal keys are stored in a Kubernetes Secret (vault-unseal-keys). Anyone who can read that Secret — through kubectl, through a compromised pod with the right ServiceAccount, through an etcd backup — can unseal Vault.

This collapses the security boundary: Vault's unseal protection becomes "Kubernetes RBAC on one Secret" instead of "cloud KMS with its own IAM policy." For a homelab, this is acceptable because:

The cluster is not exposed to untrusted users
The threat model is "protect against accidental unseal, not nation-state attacker"
Manual unseal after every restart is operationally unsustainable

For production, use KMS-based auto-unseal. For a homelab where the alternative is "Vault stays sealed until I notice," the sidecar is the pragmatic choice.

Why 5 Seconds

The initial implementation polled every 30 seconds. This created a window where Vault was sealed but the sidecar hadn't tried to unseal it yet. During that window:

ExternalSecret refresh requests failed
Authelia's init container couldn't read hmac-secret
Any service that depends on Vault-backed secrets was stuck

30 seconds of cluster-wide secret unavailability on every Vault restart. Tightening to 5 seconds reduced the seal window to an acceptable range — most Vault restarts complete unseal within 5 seconds, and the downstream impact is minimal.

The cost: the sidecar makes one vault status call and potentially three vault operator unseal calls every 5 seconds. On Vault's API, this is negligible — it's health-check-level traffic.

The ExternalSecret Dependency Chain

The real reason Vault auto-unseal matters: every ExternalSecret in the cluster depends on Vault being unsealed.

Vault sealed
  → ExternalSecret refresh fails
    → Kubernetes Secrets not updated
      → Pods using those secrets start with stale/missing data
        → Authelia can't start (no hmac-secret)
        → Open WebUI can't start (no WEBUI_SECRET_KEY)
        → Paperless can't start (no database password)

The chain reaction is invisible until you check pod logs and see secret "authelia-secrets" not found or connection refused to Postgres (because the password never synced from Vault).

Before the sidecar, I'd come back to a sealed Vault after a node restart and spend 15 minutes manually unsealing with three key shares while half the cluster sat in CrashLoopBackOff. The sidecar turned a 15-minute manual operation into a 5-second automatic one.

The Network Policy

Vault's ingress is locked down:

# kubernetes/system/vault/network-policies.yml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vault-allow-intra-namespace
  namespace: vault
spec:
  podSelector: {}
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - namespaceSelector: {}
          podSelector: {}
    - from:
        - namespaceSelector:
            matchLabels: {}
          podSelector:
            matchLabels:
              app.kubernetes.io/name: external-secrets
      ports:
        - port: 8200

Default-deny ingress on the vault namespace. Only two sources can reach Vault's API port 8200: other pods in the vault namespace (the unseal sidecar), and External Secrets Operator pods. Everything else is blocked.

This is the minimum viable network segmentation for Vault: it needs to be reachable by ESO for secret syncing and by the unseal sidecar for auto-unseal, but nothing else needs direct Vault API access.

What I'd Change

Use Vault's built-in auto-unseal with a cloud KMS if available. The sidecar is a workaround for Vault OSS limitations. If you're running Vault Enterprise or can tolerate the cost of a cloud KMS, use it. The sidecar exists because my homelab doesn't have a KMS. Vault's configuration (policies, auth roles) is a separate concern from unsealing — see the staged Terraform migration for how that part is managed without touching unseal keys.
Store unseal keys in a more secure backend. The Kubernetes Secret is the weakest link. An improvement would be to store the keys in an HSM or a separate, more restricted secret backend. But at that point, you've basically built KMS-based auto-unseal from scratch.
Add monitoring on the sidecar. An alert when Vault transitions from sealed to unseal would provide visibility into restart frequency and sidecar health.

Vault auto-unseal without KMS is the same problem as managing encryption keys in environments without HSMs: you're trading security boundary strength for operational practicality. Azure Key Vault Managed HSM provides FIPS 140-2 Level 3 key protection — but it costs money and adds complexity. For non-production environments, the polling sidecar gives you 90% of the operational benefit at 10% of the security cost. The key is knowing which trade-off you're making.

Chaos Mesh in a Homelab: Weekly Pod-Kills on a Single-Host Cluster

david — Mon, 24 Aug 2026 07:13:35 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Chaos engineering sounds like a production-only practice — inject failures, measure impact, improve resilience. But the biggest value of chaos testing isn't catching production failures. It's finding the problems you didn't know existed, in infrastructure you thought was solid, before they find you.

I run two weekly chaos experiments on my single-host k3s cluster: a pod-kill every Sunday at 03:00 UTC, and a 100ms network latency injection at 03:30 UTC. Both are scoped to pods labeled chaos-kill: enabled in the apps namespace. Here's what they've taught me.

View the complete homelab infrastructure source on GitHub 🐙

The Setup

Chaos Mesh runs in the chaos-mesh namespace, deployed via Helm chart v2.8.3:

# kubernetes/system/chaos-mesh/application.yml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: chaos-mesh
  namespace: argocd
spec:
  source:
    repoURL: https://charts.chaos-mesh.org
    chart: chaos-mesh
    targetRevision: 2.8.3
    helm:
      values:
        chaosDaemon:
          runtime: containerd
          socketPath: /run/k3s/containerd/containerd.sock

The socketPath is critical: k3s uses its own containerd socket, not the standard Docker or containerd paths. If you point Chaos Mesh at the wrong socket, pod-kill experiments silently fail — the controller reports success but no pods are actually killed.

The two Schedule resources:

# Weekly pod-kill — Sunday 03:00 UTC
apiVersion: chaos-mesh.org/v1alpha1
kind: Schedule
metadata:
  name: weekly-pod-kill
  namespace: chaos-mesh
spec:
  schedule: "0 3 * * 0"
  type: PodChaos
  podChaos:
    action: pod-kill
    mode: one
    selector:
      labelSelectors:
        chaos-kill: "enabled"
      namespaces: ["apps"]
    gracePeriod: 0
---
# Weekly network latency — Sunday 03:30 UTC
apiVersion: chaos-mesh.org/v1alpha1
kind: Schedule
metadata:
  name: weekly-network-latency
  namespace: chaos-mesh
spec:
  schedule: "30 3 * * 0"
  type: NetworkChaos
  networkChaos:
    action: delay
    mode: all
    selector:
      labelSelectors:
        chaos-network: "enabled"
      namespaces: ["apps"]
    delay:
      latency: "100ms"
      jitter: "10ms"
    duration: "5m"
    direction: to

The mode: one on the pod-kill means exactly one pod matching the label is killed per experiment. mode: all on the network latency applies to all pods with the label. The gracePeriod: 0 on pod-kill means immediate termination — no graceful shutdown, which is the realistic failure mode (a kernel panic, a power loss, a node crash).

What Broke: First Week

The first pod-kill Sunday killed one Authelia pod. Expected behavior: Kubernetes reschedules it within seconds. What actually happened:

Pod killed at 03:00:00 UTC
Kubernetes schedules replacement at 03:00:02 UTC
Replacement starts wait-for-vault-secret init container
Init container polls Vault for ExternalSecret sync at 30s intervals
Authelia fully ready at 03:01:30 UTC — 90 seconds of downtime

Ninety seconds for Authelia to recover from a pod-kill. The init container is the bottleneck: it waits for Vault to provide the hmac-secret, OIDC keys, and session secrets before the main container starts. During those 90 seconds, any service that checks Authelia for authentication returns 501 — the Traefik middleware can't reach Authelia's /api/verify endpoint.

The fix wasn't to make Authelia faster. The fix was to recognize that 90 seconds of Authelia downtime is acceptable for a single-replica-killed scenario, but unacceptable for a two-replica scenario where both pods are killed simultaneously. The PDB (minAvailable: 1) prevents simultaneous kills — Chaos Mesh respects PodDisruptionBudgets. Without the PDB, both pods could be killed in the same experiment window.

What Broke: Network Latency

The 100ms latency injection was more insidious. It didn't break anything immediately. Instead, it exposed timing-dependent behavior that was invisible under normal network conditions:

Velero backup duration increased by 40%. Velero's Kopia sidecar communicates with the Garage S3 endpoint over the cluster network. Adding 100ms per request multiplied across thousands of file operations extended the backup window from ~12 minutes to ~20 minutes.

ArgoCD sync operations became sluggish. ArgoCD's repo-server fetches manifests from the git repo, applies diffs, and syncs. Each step involves network calls that now had 100ms added. Syncs that normally took 5 seconds took 15-20 seconds.

Uptime Kuma monitors flickered. Uptime Kuma's HTTP monitors expect sub-second response times. The 100ms added by Chaos Mesh pushed some monitors past their threshold, generating false-positive "service down" alerts.

None of these are failures. They're performance degradation under adverse conditions. But they reveal the hidden assumption in every service's timeout and retry configuration: "the network is fast." When it isn't — because of a real network issue, a noisy neighbor, a congested switch — services that work fine under normal conditions start failing in unexpected ways.

What It Taught

1. PDBs aren't optional

Before Chaos Mesh, the PodDisruptionBudgets for Authelia and cloudflared were theoretical — "we have them because best practices say we should." After the first pod-kill confirmed that the PDB actually prevented simultaneous kills, they became load-bearing infrastructure.

2. Init containers are single points of failure

Every ExternalSecret-backed deployment has an init container that waits for Vault. If Vault is slow, sealed, or unreachable, the init container blocks the entire pod startup. The 90-second Authelia recovery time is entirely dominated by this init container. A faster health check or a cached secret fallback would reduce recovery time.

3. Latency exposes timeout assumptions

Every service has implicit assumptions about network latency. When those assumptions are violated, services don't crash — they degrade. The degradation is harder to debug than a crash because everything looks healthy in the logs. The only signal is slower response times and increased error rates that don't quite reach alerting thresholds.

4. Chaos testing on a homelab isn't overkill

The cluster has no SLA. Nobody is paying for uptime. But the same workloads (Postgres, Vault, Authelia) run in production environments everywhere. Finding that a pod-kill takes 90 seconds to recover — on a homelab where the consequence is "I can't log in for a minute and a half" — is infinitely better than finding it in production where the consequence is a customer-facing outage. It's the same reasoning behind treating a cascading OOM-kill/DNS-storm failure as worth a full root-cause writeup even though nobody was paged.

Chaos engineering at enterprise scale uses the same tools: Azure Chaos Studio for VM and AKS fault injection, Azure Load Testing for performance baseline, and Azure Monitor for measuring blast radius. The principle is identical — inject realistic failures in a controlled environment, measure the impact, fix what breaks. The only difference is that Azure Chaos Studio charges per experiment, so you want your homelab practice run to be thorough.

Cloudflare Tunnel Without Opening a Single Firewall Port

david — Mon, 17 Aug 2026 07:10:16 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Three services in my homelab need external access: Immich for family photo backup, Atlantis for Terraform webhooks, and Jellyfin for media. None of them have a single inbound firewall rule — the MikroTik firewall stays default-deny with no WAN-facing accept rules at all. Cloudflare Tunnel handles all external traffic through an outbound-only connection, and split-DNS on AdGuard ensures LAN clients reach services directly without leaving the network.

View the complete homelab infrastructure source on GitHub 🐙

The Architecture

Cloudflare Tunnel works by running a cloudflared daemon inside the cluster that maintains an outbound-only connection to Cloudflare's edge. External requests hit Cloudflare, get routed through the tunnel to the origin service, and back. No inbound ports, no port forwarding, no attack surface on the WAN side.

The Terraform configuration defines the tunnel ingress rules:

# terraform/stacks/cloudflare/main.tf
resource "cloudflare_zero_trust_tunnel_cloudflared_config" "homelab" {
  tunnel_id = var.tunnel_id
  account_id = var.account_id
  config {
    ingress {
      hostname = "atlantis.woitzik.dev"
      service  = "https://traefik.apps.svc.cluster.local:443"
      origin_request {
        origin_server_name = "atlantis.woitzik.dev"
        no_tls_verify     = false
      }
    }
    ingress {
      hostname = "photos.woitzik.dev"
      service  = "http://immich-server.apps.svc.cluster.local:2283"
      origin_request {
        no_tls_verify    = false
        chunked_encoding = true
      }
    }
    ingress {
      hostname = "media.woitzik.dev"
      service  = "http://ct-srv-jellyfin-01.dmz.woitzik.dev:8096"
      origin_request {
        no_tls_verify = false
      }
    }
    ingress {
      service = "http_status:404"
    }
  }
}

The catch-all http_status:404 is mandatory — Cloudflare Tunnel requires a final rule that catches unmatched hostnames.

Split-DNS on AdGuard

The tunnel creates a split-brain DNS problem. External requests go through Cloudflare's anycast IPs. Internal requests (LAN) should go directly to the Traefik VIP, bypassing Cloudflare entirely. This means photos.woitzik.dev needs to resolve to different IPs depending on where the client is.

AdGuard Home handles this with wildcard overrides:

# AdGuard rewrites
# Default: *.woitzik.dev → 10.0.20.200 (Traefik VIP)
# Override: photos.woitzik.dev → Cloudflare anycast (172.67.137.91, 104.21.38.184)

The *.woitzik.dev → 10.0.20.200 wildcard covers all internal services. The specific override for photos.woitzik.dev points to Cloudflare's anycast IPs, so mobile clients on cellular networks get routed through the tunnel while LAN clients go direct.

The gotcha: AdGuard's wildcard applies to the search domain too. If the Proxmox host's search domain is woitzik.dev, every internal hostname query goes through the wildcard — including pve.woitzik.dev, pbs.woitzik.dev, and other services that should never leave the LAN. This was caught during initial setup: PTR queries for k3s pod IPs were being forwarded to the FritzBox because AdGuard's search domain configuration leaked DNS traffic outward.

The Chunked Encoding Fix

Large photo and video uploads to Immich through Cloudflare Tunnel were failing with ECONNRESET. The uploads would complete 60-80% then drop the connection.

The root cause: Cloudflare's proxy buffers the entire request body before forwarding to the origin. For multi-GB video uploads, this exceeds Cloudflare's buffer limit and the connection resets.

The fix in the Terraform tunnel config:

origin_request {
  chunked_encoding     = true
  write_timeout        = 600  # seconds
  read_timeout         = 120  # seconds
}

chunked_encoding = true tells cloudflared to stream the request body to the origin in chunks instead of buffering the entire payload. write_timeout = 600s gives large uploads up to 10 minutes to complete.

This was the Immich-specific fix documented in CHANGELOG v0.8.0. The same issue can affect any service behind Cloudflare Tunnel that accepts large file uploads — Nextcloud, Paperless, or any service accepting multipart form data above ~100MB.

Atlantis Through Traefik

Atlantis needs to receive GitHub webhooks for Terraform PR events. The naive approach: expose Atlantis directly via Cloudflare Tunnel. The problem: Atlantis has no built-in authentication — anyone who knows the URL can trigger plans and applies.

The fix: route Atlantis through Traefik first, which applies the Authelia ForwardAuth middleware before the request reaches Atlantis:

ingress {
  hostname = "atlantis.woitzik.dev"
  service  = "https://traefik.apps.svc.cluster.local:443"
  origin_request {
    origin_server_name = "atlantis.woitzik.dev"
  }
}

The Cloudflare Tunnel points at Traefik, not Atlantis. Traefik's IngressRoute for Atlantis includes the Authelia middleware:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: atlantis
  namespace: apps
spec:
  entryPoints: [websecure]
  routes:
    - match: Host(`atlantis.woitzik.dev`)
      kind: Rule
      middlewares: [{name: authelia}]
      services:
        - name: atlantis
          port: 4141
  tls:
    secretName: wildcard-woitzik-dev-tls

GitHub webhooks bypass Authelia because the webhook source IPs are known and can be allowlisted separately. But the Atlantis web UI — the plan output, the apply buttons — requires Authelia authentication.

This pattern (Cloudflare Tunnel → Traefik → Authelia → service) is the same for every external service. The tunnel handles ingress, Traefik handles routing and TLS, Authelia handles authentication. Each layer does one thing.

DNS Record Management

Cloudflare DNS records for the tunnel CNAMEs are Terraform-managed:

resource "cloudflare_dns_record" "tunnel_photos" {
  zone_id = var.zone_id
  name    = "photos"
  type    = "CNAME"
  content = "abc123.cfargotunnel.com"
  proxied = true
  ttl     = 1
}

The proxied = true means traffic goes through Cloudflare's proxy, which provides DDoS protection, rate limiting, and WAF rules. ttl = 1 is automatic for proxied records.

The auth DNS record (CNAME to home.woitzik.dev) is proxied = false — it's an internal-only record that doesn't need Cloudflare's proxy layer. Same for the Minecraft playit.gg record.

Cloudflare Tunnel's zero-inbound-port model is the same pattern as Azure Private Link and Azure Front Door: external traffic enters through a managed proxy, never touches your firewall directly, and the origin service only needs outbound connectivity. The split-DNS complexity is the same as Azure DNS Private Zones — internal resolution goes through private endpoints, external resolution through public DNS. Different tools, identical architecture.

K3s on Raspberry Pi: Why I Said No (And You Probably Should Too)

david — Sat, 15 Aug 2026 10:03:33 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

The idea was elegant: three Raspberry Pi 5 nodes running k3s control-plane + embedded etcd, replacing the three Proxmox VMs. Lower power consumption, physical separation from the main host, and a genuine HA cluster spread across real hardware.

It didn't work. The failure wasn't dramatic — no kernel panic, no cluster death. It was a slow accumulation of fragility that made the cluster less reliable than the single-node setup it replaced.

View the complete homelab infrastructure source on GitHub 🐙

The Attempt

The plan (ADR-014, Option B): run k3s embedded etcd on three Raspberry Pi 5 (8GB)* nodes. The Pis already handle DNS (AdGuard + Unbound) and Keepalived VIP. Adding k3s control-plane seemed like a natural extension.

The k3s cluster spec:

3x Raspberry Pi 5 (8GB)
256GB microSD cards (A2 rated)
Gigabit Ethernet via USB 3.0 adapter (Pi 5's native Ethernet is limited)
k3s v1.31 with embedded etcd

Why It Failed: SD Card Fragility

etcd is write-heavy. Every Kubernetes API operation — pod scheduling, configmap updates, secret rotations — generates etcd writes. The k3s embedded etcd writes continuously to the local filesystem.

SD cards have limited write endurance. The A2-rated cards I used are rated for ~150 MB/s sequential write, but the random 4K write IOPS that etcd generates are a different story. Under sustained write load, the SD card's write amplification factor increases, the garbage collection cycle can't keep up, and write latency spikes.

The symptom: etcd response times would occasionally jump from 10ms to 500ms+ for no apparent reason. No CPU load, no network congestion, no memory pressure. Just slow disk writes. On a NVMe-backed VM, etcd writes complete in microseconds. On an SD card, they're orders of magnitude slower.

etcd has a built-in leader election timeout (default 5s). When etcd write latency exceeds the election timeout, the leader steps down and a new election starts. If the new leader is also on an SD card with the same write latency problem, the election can fail too. The result: brief periods where the Kubernetes API server is unavailable, even though all nodes are "healthy."

Why It Failed: Network Latency

The three Pis communicate over Gigabit Ethernet via USB 3.0 adapters. The adapter adds ~0.5ms of latency per hop compared to native Gigabit. For etcd consensus, which requires a majority of nodes to acknowledge each write, the added latency compounds:

Write to leader: 0.5ms
Leader replicates to 2 followers: 0.5ms × 2 = 1ms
Followers acknowledge: 0.5ms × 2 = 1ms
Total round-trip: ~2.5ms

Compare to three VMs on the same NVMe-backed host: ~0.1ms round-trip. The 25x latency increase doesn't matter for normal API operations, but it matters during high-write periods or leader elections, exactly when low latency is most critical.

Why It Failed: Resource Contention

The Pis were already running AdGuard + Unbound for network DNS, Keepalived for the VIP, and node_exporter for monitoring. Adding k3s control-plane + etcd meant four services competing for the same CPU, memory, and — critically — the same SD card.

When AdGuard's DNS cache expired and refreshed simultaneously with an etcd compaction cycle, both hit the SD card at once. The resulting I/O contention produced the same pattern as the Proxmox host freeze: etcd write latency spikes → leader election → temporary API unavailability.

The Deliberate Reversion

After three weeks of monitoring, the k3s-on-Pi cluster was less reliable than the single-node setup it replaced. The decision (ADR-014): revert to a single k3s control-plane on the Proxmox VM (vm-srv-k3s-11), with two agent-only worker nodes.

# vm-srv-k3s-11: control-plane + etcd (sole server)
# vm-srv-k3s-12: agent only
# vm-srv-k3s-13: agent only

The single-server setup has a known limitation: if k3s-11 goes down, the entire cluster is down. No failover, no HA. But "down" in a homelab context means "inaccessible for a few minutes while I restart the VM" — not "data center outage affecting thousands of users."

The trade-off: reliability (single NVMe-backed VM) over availability (three SD-card-backed Pis). For a homelab, reliability is the right priority.

What the Pis Do Instead

The Raspberry Pis remain in the rack, running DNS and Keepalived — exactly what they're good at:

AdGuard Home: DNS filtering for the entire network, low write volume, well within SD card endurance
Unbound: Recursive DNS resolver, almost entirely read operations after cache warm-up
Keepalived: Active/passive VIP failover, heartbeat-only, negligible I/O
node_exporter: System metrics, read-only

These workloads are low-write, low-latency-sensitive, and can tolerate brief interruptions without affecting the cluster. The Keepalived VIP failing over to the other Pi is a 1-second blip. The k3s API server going down for 30 seconds during an etcd election breaks every kubectl command and ArgoCD sync.

The General Lesson

etcd's requirements are specific: low-latency, high-endurance storage with consistent write performance. NVMe provides this. SATA SSDs provide this (mostly). SD cards and USB-attached storage do not, because their write latency is unpredictable under sustained load.

If you're running k3s on Raspberry Pis, the stable configuration is: Pis as agent-only workers, control-plane on a VM or dedicated x86 box with proper storage. The Pis handle DNS, monitoring, and lightweight workloads — things where an SD card's write endurance is adequate and a brief interruption doesn't cascade into a cluster-wide event.

The same logic applies to etcd on cloud VMs: don't put etcd on Standard HDD. The IOPS and latency guarantees of Premium SSD or Ultra Disk exist specifically because etcd's consensus protocol requires consistent, low-latency writes. The Pi's SD card is the cloud equivalent of a Standard HDD — it works until you need consistent performance under load.

K3s on ARM is production-ready for worker nodes. The control-plane decision is about storage performance and write consistency, not CPU architecture. Azure's AKS control plane runs on the same principle: the control-plane nodes use Premium SSD managed disks precisely because etcd needs consistent, low-latency writes. The underlying hardware doesn't matter — the I/O guarantees do.

Azure Logic App Standard: Private Storage Needs Four Private Endpoints

david — Sat, 15 Aug 2026 09:51:09 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Logic App Standard is the go-to hosting model when your workflows need VNet integration and a predictable App Service Plan. In a hardened Azure landing zone, that means one thing: a Storage Account that is not reachable from the public internet, fronted by Private Endpoints.

The deployment succeeded. The Resource Group, the Storage Account, the private endpoints, the DNS zones - everything planned cleanly and applied without a single error. Then the workflow host refused to start. Every single boot ended in the same 403 Forbidden, roughly 600 milliseconds after the host registered its blob webhook endpoint.

This is the postmortem: why it happened, why four plausible fixes did not help, and how the generic rebuild now bakes the correct configuration in by default.

View the base source code on GitHub 🐙

The Symptom

The portal designer showed a generic failure - renderComponentIntoRoot, Workflow validation failed - none of which pointed anywhere useful. A direct API call against the host runtime revealed the real error:

Encountered an error (ServiceUnavailable) from host runtime.

The Kudu host status endpoint (/hostruntime/admin/host/status) confirmed the host was not just having a bad moment:

{
  "state": "Error",
  "errors": [
    "Microsoft.WindowsAzure.ResourceStack: Unexpected HTTP status code 'Forbidden'. The remote server returned an error: (403) Forbidden."
  ]
}

The Forbidden came from Storage itself, and it came back fast. That timing was the first real clue: no timeout, no firewall blackhole, no DNS hang - a prompt, deliberate rejection.

The Root Cause

A Logic App Standard is a Functions host under the hood - a functions app hosting workflow apps. Its internal host runtime does not just need Storage for your workflow content. It consumes four Storage subresources during normal operation:

Subresource	Used for
File	Content share - `host.json`, `connections.json`, workflow definitions
Blob	Extension-bundle cache, application logs
Queue	Internal WebJobs coordination - scale controller, trigger bookkeeping
Table	Internal WebJobs metadata

Our wrapper built private endpoints for Blob and File - the two that look obviously necessary, because the File share literally holds the content and Blob holds the bundle cache. Queue and Table were forgotten.

The consequence is subtle and nasty. DNS for *.queue.core.windows.net and *.table.core.windows.net resolved to the public IP - there is no private endpoint, so no private DNS record. But the Storage Account had publicNetworkAccess disabled. Every request the host made to Queue or Table over the public IP was rejected by Storage itself with 403 Forbidden - instantly. That is why the failure appeared exactly ~600 ms after boot: Storage denies fast, it does not hang.

Why Four Plausible Fixes Failed

Each attempt was reasonable, and each targeted the wrong layer:

RG Contributor for the Managed Identity. RBAC was never the problem. The Storage network firewall does not consult RBAC roles - it only evaluates network origin plus keys/auth. Adding permissions changed nothing.
vnet_route_all_enabled = false. This flag only controls whether outbound internet traffic is routed through the VNet. Queue/Table calls never went through the VNet in the first place - they were direct public calls caused by wrong DNS resolution. The flag was for a different layer of the problem.
Storage data-plane RBAC (Blob/Queue/Table Data Contributor). Permission is useless when the request is rejected at the network layer before RBAC is even evaluated.
Finally, Application Insights. The Kudu logs showed only the short error with no stack trace. Attaching App Insights temporarily revealed the full exception stack and the exact failing method (WorkflowExtensionProvider.Initialize). Only then was the real dependency visible.

The lesson from the four failed fixes: when a private Storage Account rejects a request with 403, the problem is almost never identity. It is reachability - specifically, which DNS name resolves to which IP.

The Fix

Two additional Private Endpoints on the same Storage Account, plus their Private DNS zone integration:

pe-st-queue → privatelink.queue.core.windows.net
pe-st-table → privatelink.table.core.windows.net

The private DNS zones already existed centrally; they just were not attached to this Storage Account. Once Queue and Table resolved to private IPs, the host booted cleanly on the next restart.

The memory anchor for next time: a Logic App Standard or Function App with private Storage needs all four Storage subresources behind Private Endpoints - not just the ones that look content-related. Queue and Table are pure host-internal infrastructure. They show up in no obvious configuration, which is exactly why they get forgotten.

The Generic Rebuild

Instead of leaving the fix in a wrapper module, the generic template now creates all four by default. The private endpoints are a single for_each over the subresource list:

locals {
  storage_subresources = ["blob", "file", "queue", "table"]
}

resource "azurerm_private_endpoint" "storage" {
  for_each            = toset(local.storage_subresources)
  name                = "pe-${local.storage_name}-${each.key}"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
  subnet_id           = azurerm_subnet.private_endpoints.id

  private_service_connection {
    name                           = "psc-${local.storage_name}-${each.key}"
    private_connection_resource_id = azurerm_storage_account.main.id
    subresource_names              = [each.key]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name                 = "dns-${each.key}"
    private_dns_zone_ids = [azurerm_private_dns_zone.main[each.key].id]
  }
}

Adding a sixth subresource (e.g. dfs for Data Lake) is now one list entry instead of a copy-pasted resource block.

The DNS zones are created and linked to the VNet in the same pass, so *.queue.core.windows.net resolves privately without touching your DNS infrastructure:

locals {
  private_dns_zones = {
    blob  = "privatelink.blob.core.windows.net"
    file  = "privatelink.file.core.windows.net"
    queue = "privatelink.queue.core.windows.net"
    table = "privatelink.table.core.windows.net"
    sites = "privatelink.azurewebsites.net"
  }
}

resource "azurerm_private_dns_zone" "main" {
  for_each            = local.private_dns_zones
  name                = each.value
  resource_group_name = azurerm_resource_group.main.name
}

resource "azurerm_private_dns_zone_virtual_network_link" "main" {
  for_each              = local.private_dns_zones
  name                  = "link-${azurerm_virtual_network.main.name}"
  resource_group_name   = azurerm_resource_group.main.name
  private_dns_zone_name = azurerm_private_dns_zone.main[each.key].name
  virtual_network_id    = azurerm_virtual_network.main.id
}

Two further details worth stealing from the fix:

1. Order the dependency. The Logic App must not boot before its private endpoints exist, otherwise the first scale-up races the network configuration:

resource "azurerm_logic_app_standard" "main" {
  # ...

  public_network_access     = "Disabled"
  virtual_network_subnet_id = azurerm_subnet.integration.id

  app_settings = merge(
    {
      "WEBSITE_CONTENTOVERVNET" = "1"
      "WEBSITE_VNET_ROUTE_ALL"  = "1"
    },
    var.app_settings
  )

  depends_on = [
    azurerm_private_endpoint.storage["blob"],
    azurerm_private_endpoint.storage["file"],
    azurerm_private_endpoint.storage["queue"],
    azurerm_private_endpoint.storage["table"],
  ]
}

WEBSITE_CONTENTOVERVNET = 1 tells the host to serve its own content share over the VNet instead of the public file endpoint.

2. Bootstrap the file share. A freshly deployed Standard Logic App boots against an empty share, and the workflow host and the portal designer both trip over that. The template uploads a minimal host.json / connections.json into site/wwwroot during the first apply - empty definition, but a valid host contract:

resource "local_file" "host_json" {
  filename = "${path.module}/.bootstrap-content/host.json"
  content = jsonencode({
    version = "2.0"
    extensionBundle = {
      id      = "Microsoft.Azure.Functions.ExtensionBundle.Workflows"
      version = "[1.*, 2.0.0)"
    }
  })
}

The Checklist

When you wire a Logic App Standard or Function App to a private Storage Account:

File private endpoint - content share
Blob private endpoint - bundle cache and logs
Queue private endpoint - host-internal coordination
Table private endpoint - host-internal metadata
Sites private endpoint on the Logic App itself (privatelink.azurewebsites.net)
Private DNS zones for all five, linked to the VNet
public_network_access = "Disabled" on the Storage Account and the Logic App
WEBSITE_CONTENTOVERVNET = 1 so content is served over the VNet
depends_on from the Logic App to the private endpoints

The Takeaway

A private endpoint per Storage subresource is not a nice-to-have for Logic Apps Standard - it is a hard requirement of the host runtime. The trap is that Queue and Table are invisible in every configuration file and every blog tutorial, so they get skipped until a 403 Forbidden starts appearing in Kudu logs at every boot.

The good news: the fix is deterministic, and it is now encoded in the template rather than remembered by an engineer who already paid the price once.

DEV Community: david

NFS vs local-path: The SQLite Trap That Corrupted My S3 Metadata

The Storage Classes

Why SQLite and NFS Don't Work

The Garage Incident

The Recovery

The Fix

The Trade-off

My Media Stack Lives in Two Containers and a Python CronJob

Why Kubernetes Was Wrong for Media

GPU Passthrough

The Traffic-Isolation Rethink

NFS File Locking

The Architecture

The Traefik Bridge

The Python CronJob

What I'd Change

Private AKS on Azure: No Public API Server, No Public Node IPs, No Default Outbound

Target Architecture

Step 1: The API Server Has No Public FQDN

Step 2: userDefinedRouting — the Setting That Actually Matters

Step 3: ACR and Key Vault Without a Public Endpoint

What This Doesn't Do

Staggered VM Boot: How I Prevented a Load Average of 147

The Boot Storm

The Fix: Staggered Boot Order

Why NFS First

The CPU Scheduling Priority

The onboot Gotcha

Before vs. After

The Overcommit Guard: How a Python Script Prevents Host Freezes

The Two Memory Physics

VM dedicated — Real Reservation

LXC dedicated — Soft Ceiling

The Two-Tier Guard

Hard Gate (CI-Fail)

Soft Check (Warn-Only)

Why floating Is Ignored

The CI Integration

What It Prevents

GitOps for Firewall Rules: MikroTik + Terraform + Atlantis

The Provider Setup

The VLAN Matrix

Deterministic Firewall Ordering

The Atlantis Flow

What Goes in Terraform vs. What Doesn't

The LED Night-Mode

The Anti-Patterns

When Your LLM Hallucinated Your OCR

The Configuration Error

How Many Documents Were Affected

The Fix

The Ollama iGPU Problem

The Lesson: Two Failure Modes Stacked

Vault Auto-Unseal Without Cloud KMS: The Polling Sidecar Pattern

The Sidecar

The Security Trade-off

Why 5 Seconds

The ExternalSecret Dependency Chain

The Network Policy

What I'd Change

Chaos Mesh in a Homelab: Weekly Pod-Kills on a Single-Host Cluster

The Setup

What Broke: First Week

What Broke: Network Latency

What It Taught

1. PDBs aren't optional

2. Init containers are single points of failure

3. Latency exposes timeout assumptions

4. Chaos testing on a homelab isn't overkill

Cloudflare Tunnel Without Opening a Single Firewall Port

The Architecture

Split-DNS on AdGuard

The Chunked Encoding Fix

Atlantis Through Traefik

DNS Record Management

K3s on Raspberry Pi: Why I Said No (And You Probably Should Too)

The Attempt

Why It Failed: SD Card Fragility

Why It Failed: Network Latency

Step 2: `userDefinedRouting` — the Setting That Actually Matters

The `onboot` Gotcha

VM `dedicated` — Real Reservation

LXC `dedicated` — Soft Ceiling

Why `floating` Is Ignored