DEV Community: david

Velero Backs Up to the Cluster It's Backing Up: Breaking the Backup Circularity

david — Mon, 19 Oct 2026 12:38:54 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

The circularity: Velero backs up into Garage S3, which runs inside the Kubernetes cluster it's backing up. Cluster lost → Garage PVC lost → Velero backup bucket gone → nothing to restore from.

This is the backup equivalent of storing your offsite backup key in the safe you're backing up. The offsite target exists in config but isn't actually reachable — and you don't realize the gap until you need it.

View the complete homelab infrastructure source on GitHub 🐙

The Two-Problem Stack

Two issues compound into structural fragility:

Problem 1: Backup Circularity

Velero's backup storage location points to Garage, an S3-compatible object store running as a StatefulSet inside the cluster. Every Velero backup — daily snapshots of Vault, Authelia, databases, ArgoCD configs — lands in a bucket that depends on the cluster surviving.

Cluster lost → Garage PVC lost → Velero backup bucket gone → nothing to restore from

PBS backs up the VM/LXC disks to a Seagate 2TB External HDD*, but PBS runs on the same physical host — a BMAX Mini PC*. Simultaneous loss of host + USB HDD means total unrecoverability at the VM layer. Velero was the last safety net — except it was circular.

Problem 2: Torn-State Hazard

Two PVCs with different storage backends, different failure modes:

PVC	Storage	Location	Failure mode
`garage-meta`	`local-path`	k3s node	Node-pinned, survives if node survives
`garage-data`	NFS (`nfs-client`)	NFS LXC	Survives if NFS LXC survives

On 2026-07-12, an unclean Garage shutdown (OOM kill) left db.sqlite corrupted — the same NFS/SQLite locking incompatibility that eventually forced every embedded database off NFS. The metadata — SQLite, bucket/key ownership, access grants — and the actual object data were on different volumes with different rollback fates. A partial restore or node loss would leave one without the other, with no clean way to reconcile them.

The Four Options

Option A: Consolidate meta + data on one volume

Move garage-meta from local-path to NFS (same backend as garage-data). Both PVCs on the same storage class → same snapshot/restore fate.

Verdict: Fixes torn-state but not circularity. NFS for SQLite is proven fragile — the 07-12 corruption happened on NFS-backed data. Moving data to local-path requires hardware you don't have yet.

Option B: External off-cluster target (Cloudflare R2)

Activate the existing r2-offsite backup storage location. Velero dual-targets: in-cluster Garage for speed + off-cluster Cloudflare R2 for bootstrap.

Cluster lost → Garage gone → Velero still has offsite copy on R2
Bootstrap: rebuild cluster → ArgoCD → Velero → restore from R2

Cost: ~$0.015/GB/month on R2 (egress free). At current backup size (~5-10Gi), negligible. Risk: Near-zero — additive only, no existing backup path changes.

Verdict: Directly solves circularity. The existing R2 config is ready — needs credentials and a "go".

Option C: PBS as the primary recovery layer

Lean into PBS as the from-scratch recovery path. PBS already backs up all VMs/LXCs (including the k3s VMs that carry Garage's data). Velero becomes a convenience layer for single-app restore, not the bootstrap path.

Verdict: Already the real bootstrap path (DR Tier 1-4). Velero is Tier 5, only if Garage survived. PBS is the actual safety net. The circularity is a Velero-level problem, not a PBS-level one.

Option D: Combine B + C (what I chose)

Activate R2 offsite for Velero (breaks circularity) AND acknowledge PBS as the primary VM-level recovery layer. Velero becomes the single-app/namespace convenience tool with a real offsite target.

Layer	What it covers	Recovery path
PBS	VM/LXC disks, full host recovery	Restore VMs from PBS → cluster up
Velero + R2	App-level config, secrets, databases	ArgoCD → Velero → restore from R2
Garage rebuild	If PVCs are lost entirely	Terraform re-creates buckets, ESO re-seeds keys

Verdict: Covers both problems. R2 breaks circularity (zero risk, additive). PBS handles VM-level recovery (already there). Garage rebuild from scratch is the fallback if PVCs are lost — better than a partial restore that could leave metadata/data inconsistent.

The Recovery Model

Three independent layers, each covering a different failure scenario:

Scenario                          → Recovery path
─────────────────────────────────────────────────────
Single app crash                  → Kubernetes restart (self-heals)
Single PVC lost                   → Velero restore from R2 offsite
Full cluster lost (host survives) → PBS restores k3s VMs → Velero from R2
Host + USB HDD lost               → Insurance (nothing saves this)

The key insight: no single layer has to be perfect. Each layer covers the failure modes the layer below misses. Garage's torn-state problem becomes irrelevant when the recovery procedure is "rebuild from scratch" rather than "restore from backup."

Implementation (Zero Downtime)

The R2 activation was purely additive — no existing backup path changed:

Create Cloudflare R2 bucket (free tier)
Populate r2-backuplocation.yml with real R2 credentials via External Secrets Operator
Apply the daily-offsite Velero Schedule (already committed)
Run one test backup, verify with velero backup describe
Confirm PodVolumeBackup entries present for PVCs

The in-cluster Garage target stays. Velero now writes to both: Garage for fast local restores, R2 for bootstrap-from-scratch. If Garage is gone, R2 has everything.

What I'd Change

Implement R2 sooner. The config existed for weeks before activation. The circularity was a known risk — the "I'll do it later" mindset is what creates disaster recovery gaps.
Add a Velero backup age alert. Alert when the most recent successful backup is older than 48 hours. If backups silently stop, you want to know before you need one.
Test the full recovery path. PBS restore of k3s VMs → cluster boot → ArgoCD reconciliation → Velero restore from R2. An untested recovery path is a theory, not a plan.

Backup circularity is the most common structural flaw in self-hosted Kubernetes: the tool that protects you depends on the thing you're protecting. The fix is always the same — add an independent offsite layer. Cloudflare R2 costs nothing at homelab scale, and the R2 egress-free model means you're not penalized for actually using the restore. The hardest part isn't the implementation. It's admitting your current backup doesn't survive the scenario you built it for.

Further reading: For understanding the full recovery architecture, Designing Data-Intensive Applications by Martin Kleppmann* covers replication, backup strategies, and the failure modes that make torn-state hazards so dangerous. The Velero circularity is a specific case of the general problem Kleppmann describes: systems that depend on their own recovery path.

The DNS Drift That Wasn't DHCP: How Proxmox Cloud-Init Silently Injects Search Domains

david — Fri, 16 Oct 2026 09:41:18 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Every DNS lookup from every pod in my k3s cluster was appending home.lan to the query. External lookups like registry-1.docker.io became registry-1.docker.io.home.lan, failed locally, recursed all the way out, and added ~341ms of latency to every image pull, every API call, every webhook — traffic that should have gone straight to Unbound for recursive resolution, not round-tripped through a phantom search domain. I assumed DHCP was the source, wrote a Terraform fix, and watched the plan show zero diff against live state.

The real source was Proxmox's cloud-init generator — three layers above the DHCP server.

The Symptom

Prometheus showed p99 DNS latency at 341ms for external queries. Local queries (services within the cluster) resolved in under 1ms. The gap was consistent across all three k3s nodes.

# PromQL showing the split
quantile_over_time(0.99, dns_request_duration_seconds{domain!~".*cluster.local"}[5m])
# → ~341ms

quantile_over_time(0.99, dns_request_duration_seconds{domain=~".*cluster.local"}[5m])
# → ~0.8ms

The kubelet's DNS policy appends the node's search domain to every pod's /etc/resolv.conf. If the node has search home.lan, every pod inherits it. With the default ndots:5, any name with fewer than 5 dots gets the search domain appended first — meaning almost everything.

Wrong Hypothesis: DHCP

My first guess was the RouterOS DHCP server. The network was configured via Terraform, but I'd never explicitly set the domain field:

# Before fix — no domain attribute at all
resource "routeros_ip_dhcp_server_network" "vlan_networks" {
  for_each   = local.homelab_vlans
  address    = "10.0.${each.value}.0/24"
  gateway    = "10.0.${each.value}.1"
  dns_server = "10.0.20.5"
}

Adding domain = "" should force Terraform to detect and correct any drift:

resource "routeros_ip_dhcp_server_network" "vlan_networks" {
  for_each   = local.homelab_vlans
  address    = "10.0.${each.value}.0/24"
  gateway    = "10.0.${each.value}.1"
  dns_server = "10.0.20.5"
  domain     = ""  # explicitly empty — should fix drift
}

The plan showed zero changes. Live state already matched domain = "". DHCP was never the source.

Live Verification

Confirmed via the RouterOS REST API:

# Check all DHCP server config paths
curl -s "https://router.lan/rest/ip/dhcp-server" | jq '.[].domain'
# → all null

curl -s "https://router.lan/rest/ip/dhcp-server/network" | jq '.[].domain'
# → all null

curl -s "https://router.lan/rest/ip/dns" | jq '.domain'
# → null

The RouterOS DHCP server had no domain/search config anywhere. domain = "" matched live state because the live state genuinely was empty. DHCP was never handing out home.lan.

The Real Source: Proxmox Cloud-Init

On the k3s nodes themselves:

networkctl status --json | jq '.DNS'
# → {"SearchDomains": ["home.lan"], "ConfigSource": "static"}

ConfigSource: static — not DHCP. The search domain was baked into the node's netplan config, generated by Proxmox's cloud-init writer.

When you create a VM in Proxmox with Terraform and leave initialization.dns.domain unset, Proxmox falls back to the PVE node's own configured search domain:

# On the Proxmox host
pvesh get /nodes/pve-mgmt-01/dns
# → search: home.lan

Proxmox's cloud-init generator reads the PVE node's DNS config and writes it into the VM's /etc/netplan/50-cloud-init.yaml. Every k3s VM got home.lan as its search domain — not from DHCP, but from the hypervisor itself.

Why This Is a Trap

The trap is the layering:

Proxmox PVE node DNS config
    ↓ (cloud-init fallback)
VM netplan search domain
    ↓ (kubelet DNS policy)
Pod /etc/resolv.conf search domain
    ↓ (ndots:5)
Every DNS lookup appends .home.lan
    ↓
~341ms latency on every external query

And the debugging trap: terraform plan showed zero diff on the DHCP resource because DHCP wasn't the source. A Terraform-native investigation would have concluded "no drift detected" and moved on. The real fix required:

Clearing the PVE node's default: pvesh set /nodes/pve-mgmt-01/dns -search ""
Regenerating cloud-init on each k3s VM
Declaring domain = "" in the Proxmox VM's initialization.dns block

Steps 1 and 2 are live changes to etcd-member VMs — no Terraform resource models this. Step 3 declares intent for future VMs but has no effect on already-provisioned ones (the dns block is in lifecycle.ignore_changes).

The Terraform Fix (Future VMs)

# terraform/stacks/proxmox/vm.tf
initialization {
  dns {
    domain  = ""  # explicitly unset — prevents PVE node fallback
    servers = ["10.0.20.5"]
  }
}

This ensures new VMs don't inherit the PVE node's search domain. Existing VMs need the manual cloud-init regen.

Lessons

1. DNS config has multiple sources. DHCP is the obvious one, but cloud-init, node-level config, and kubelet policy all contribute. Check all layers.

2. terraform plan zero-diff doesn't mean no drift. It means the specific resource you checked matches state. The drift might be in a different layer entirely.

3. networkctl status --json is your friend. The ConfigSource field tells you whether the search domain came from DHCP, static config, or another source. If it says "static", stop looking at DHCP.

4. Proxmox cloud-init is a hidden config source. If you leave initialization.dns.domain unset, Proxmox fills it from the PVE node. Always declare it explicitly. It's the same category of hidden-layer surprise as Proxmox's onboot attribute silently not round-tripping through Terraform — the provider abstraction has gaps that only show up once you go looking for drift that isn't there.

Further reading: DNS debugging keeps coming back to the same fundamentals — search domains, ndots, resolution order. Netzwerke: Verstehen, Einrichten, Administrieren* covers those fundamentals in more depth than this Proxmox-specific drift case gets into.

DNS latency from inherited search domains is invisible in most monitoring setups. You'll see high p99 latency on external queries but no obvious cause. If your pods have ndots:5 (the default) and your nodes have a search domain you didn't explicitly set, you're paying the tax on every lookup.

k3s vs Docker Swarm vs Nomad in 2026

david — Fri, 16 Oct 2026 09:40:43 +0000

Choosing between k3s, Docker Swarm, and Nomad stopped being a features question some time ago. It is now a project-trajectory question, and the three projects are on genuinely different trajectories. k3s tracks upstream Kubernetes within weeks and ships three supported release lines at once. Docker Swarm still works but its vendor has redirected development toward Kubernetes, and the Docker Engine v29 API floor broke parts of it badly. Nomad is the technically lightest option and the only one of the three that is not open source at all.

TL;DR — The Picks

Orchestrator	Pick it if	Main risk in 2026	License
k3s	You want Kubernetes, or your tooling already assumes it	You inherit the Kubernetes upgrade treadmill	Apache-2.0
Docker Swarm	You already run it and need cluster mode with no extra daemon	Docker Engine v29 dropped client and plugin compatibility	Apache-2.0
Nomad	You want one scheduler for mixed container and non-container workloads	BUSL source-available, not open source	BUSL → MPL-2.0 after 4 years

If you are starting fresh in 2026, the honest default is k3s. Not because it is the most capable — Nomad is lighter and Docker Swarm is simpler to learn — but because it is the only one of the three with a release cadence you can plan against, and because the surrounding ecosystem has consolidated around Kubernetes.

The three facts that decide it

Before features: these are the three things that actually constrain the decision, and all three are documented rather than opinion.

1. Docker Engine v29 broke Swarm compatibility

Docker Engine v29 raised the minimum supported daemon API version to 1.44, dropping all compatibility with clients built against Docker Engine older than v25. There is no fallback and no negotiation: tooling compiled against an older API receives a hard rejection.

The damage is not evenly distributed, and this is the part that matters if you run Swarm. Legacy (V1) Docker volume plugins are rejected by the v29 daemon. Because Swarm requires cluster-scoped persistent storage, Swarm operators hit this disproportionately. The failure mode is nasty: Docker keeps the volume metadata but cannot reach the plugin needed to mount the volume, so your data appears to be gone while it is still intact on the storage target.

There is a second trap in the same release. Upgrading some cluster nodes to v29 while others stay older — a staged rollout, the obvious upgrade strategy — causes encrypted overlay traffic to silently stop between v29.0.0 and v29.2.0. Silent is the operative word. If you use encrypted overlay networks, you must move all swarm hosts to v29.1.2 or later in one step; rolling upgrades are not available to you.

2. Docker has redirected Swarm development toward Kubernetes

Docker's own retired-products page is unusually direct about SwarmKit, the project that powers Swarm mode:

That is the vendor stating the trajectory. Swarm is not deprecated in the sense of being scheduled for removal — docker swarm init still works and the mode is still documented as built into the Engine — but it is a maintenance posture, not an investment posture. Portainer, the most visible Swarm management platform, published an advisory on 2026-03-10 recommending caution with continuing to run Swarm and planning a gradual move to Kubernetes.

3. Nomad is not open source

This one gets repeated wrong. Nomad Community Edition is licensed under the Business Source License. The change license is MPL 2.0, but it applies four years after the version is published. So a Nomad release from today becomes MPL 2.0 in 2030 — meaning that today, in 2026, if you need a genuinely open orchestrator, Nomad is not one.

The licensing situation is now stable but newly complicated. Since April 2026 HashiCorp's self-managed products follow IBM's enterprise software lifecycle, following the IBM acquisition. Nomad's support windows:

Version	Base support ends	Extended support ends	Ongoing extended ends
1.8.x (LTS)	2026-04-30	2026-04-30	2026-04-30
1.10.x (LTS)	2027-04-30	2027-04-30	2027-04-30
1.11.x	2026-10-31	2026-10-31	2026-10-31
2.0.x	2028-04-30	2029-04-30	2032-04-30

Note that 1.8.x is already fully out of support as of this writing, and 1.11.x ends 2026-10-31 — within weeks. Nomad v1.10.x is the last LTS line. From the 2.0.0 release onward, Nomad also drops semantic versioning for IBM's Version-Modification-Fix model, so tooling that parses Nomad version strings needs updating.

One operational warning worth knowing before you plan a mixed estate: a Nomad Enterprise cluster cannot be downgraded, and servers running open-source Nomad will panic if joined to an Enterprise cluster, because the Raft entries for Enterprise features are not parseable by the open-source build. Test that boundary before you need it.

k3s — the default for a reason

k3s is a conformant Kubernetes distribution packaged as a single binary. As of October 2026 it maintains three release lines concurrently: v1.36.4+k3s1 is current, with v1.35.8 and v1.34.11 still receiving backports. Each build pins its whole dependency set — for v1.34.11+k3s1 that is Kubernetes v1.34.11, Kine v0.16.4, SQLite 3.53.4, etcd v3.6.14-k3s1, containerd v2.2.7-k3s1, runc v1.4.2, Flannel v0.28.4, metrics-server v0.9.0, Traefik v3.7.13, CoreDNS v1.14.7, helm-controller v0.16.27 and local-path-provisioner v0.0.37.

That table matters more than it looks. When something breaks at 02:00 you can read which exact version of which component is running instead of discovering it. My own homelab runs k3s across several nodes, and that pinned-component view is what makes problems like ArgoCD cache staleness on a shared resource or picking the wrong storage backend for cluster state diagnosable at all.

The cost of k3s is that you inherit Kubernetes' upgrade treadmill. Upgrading three majors is not a hobby, and every Helm chart in your stack is a chance for the chart to be behind the API version you just moved to. One concrete example from the 1.34 line: the bundled Traefik chart moved to v40.x, which renamed the ingress-nginx migration provider from kubernetesIngressNginx to kubernetesIngressNGINX. That is a breaking change inside a patch-level chart bump.

I have written about the parts that bite in practice — the Velero and Garage S3 backup chain and Unbound with AdGuard and keepalived are both k3s deployments that took real iteration to get right.

Docker Swarm — simpler, and now on borrowed time

Swarm's genuine advantage is that it is already there. No second daemon, no new control plane, no new YAML dialect. If your Docker workflow is a compose file and a handful of services, docker stack deploy is a very small step up, and the service model with rolling updates and health checks is more capable than plain Compose.

The case against it in 2026 is the two items above: the v29 compatibility break with its silent overlay-traffic failure mode, and a vendor that has said in writing that development has slowed. Neither makes Swarm unusable tomorrow. Both mean every future upgrade is a risk you cannot get support for.

If you are on Swarm, the sensible 2026 posture is: pin the Engine version deliberately, verify you are past v29.1.2, audit for V1 volume plugins before you upgrade rather than after, and start introducing Kubernetes for new workloads so you are not migrating everything at once.

Nomad — the best scheduler, the weakest license

Setting licensing aside, Nomad is genuinely well-designed for its niche. It schedules not just containers but binaries, VMs and batch jobs through one job specification, it is built around a single deliberately small binary rather than a multi-component control plane, and its UI is easier to reason about than anything in the Kubernetes ecosystem. For a mixed estate — a few containers alongside a scheduled data job, an ETL step, a Windows task — that unification is the whole point and Kubernetes does not do it without extra machinery.

Against that: BUSL rather than an OSI license, a support matrix reshuffled by the IBM acquisition, a semantic-versioning break at 2.0, and an ecosystem that assumes Kubernetes. If your team already lives in Kubernetes YAML, Nomad's job syntax is a second language to maintain for a saving you have to measure yourself.

Head-to-Head

Dimension	k3s	Docker Swarm	Nomad
Scheduling unit	Pod	Service (task)	Job (group, task, service)
Non-container workloads	No	No	Yes — binaries, VMs, batch
Control plane footprint	Single binary per node	Built into the Docker Engine, no extra daemon	Single binary
License	Apache-2.0	Apache-2.0	BUSL, becomes MPL-2.0 after 4 years
2026 release cadence	Three concurrent K8s lines	Slowed, vendor-prioritised elsewhere	Spring/fall features, monthly as needed
Ecosystem direction	Consolidating around it	Maintenance	Stable, narrower
Upgrading is	Hard (K8s treadmill)	Easy	Moderate (mind the VMF versioning break)

Which one should you run?

Run k3s if your workloads are containers and you want the Kubernetes ecosystem — Helm, ArgoCD, Prometheus operators, the CNCF ecosystem at large. This is the default in 2026 and the choice I would make for anything new.

Stay on Docker Swarm only if you already run it in production and the migration cost is real. Treat it as a managed liability: pin versions, avoid the v29 trap, and plan the exit. Do not adopt it new in 2026.

Run Nomad if scheduling heterogeneous workload types on one control plane is worth more to you than an OSI license and a Kubernetes-shaped ecosystem. Just make that trade knowingly — you are choosing source-available software with a four-year conversion clock, not open source.

If you genuinely cannot decide between two, run both on the same hardware for a month with one real workload. The theoretical comparison above is the least interesting part; how each behaves with your actual data on your actual disks is what decides it.

Frequently Asked Questions

Is Docker Swarm being deprecated?

Not formally in the sense of a scheduled removal — docker swarm init still works and Swarm mode is still part of Docker Engine. But Docker's own documentation states that SwarmKit development "has slowed in favor of Kubernetes-based solutions," so treat it as maintenance-only. The practical risk arrived with Docker Engine v29, which broke client and plugin compatibility for Swarm disproportionately.

Is Nomad open source?

No. Nomad Community Edition is licensed under the Business Source License. Versions become MPL 2.0 four years after publication, so a release from 2026 becomes genuinely open source in 2030. If you require an OSI-approved license today, that rules Nomad out. k3s and Docker Swarm are both Apache-2.0.

Is k3s just "Kubernetes but worse"?

No. It is a conformant Kubernetes distribution — same API server, same controllers, same ecosystem — delivered as one binary with sensible defaults. The compromise is operational, not functional: you give up control over the component versions to get a maintainable install, which is why the pinned component table is worth reading before you debug anything.

Can I run Docker Swarm and Kubernetes on the same host?

Technically, yes, but you should not. Both want to own the container runtime and the network stack, and the interaction between a Swarm overlay network and a CNI is a debugging experience you do not need to have. Run them on separate machines.

What is the safest way to upgrade a Swarm cluster to Docker v29?

Get every host to v29.1.2 or later in a single step. Staged rollouts break encrypted overlay traffic silently in v29.0.0 through v29.2.0. Before you start, inventory your volume plugins — any V1 plugin will be rejected by the v29 daemon, and the resulting "missing" data is usually still on the storage target. Export your data first.

Does Nomad support Kubernetes workloads?

Not natively. Nomad is not a Kubernetes scheduler and does not run Kubernetes manifests. There is no compatibility path if you need the Kubernetes API; pick k3s instead. Nomad's own strength — scheduling binaries and batch jobs alongside containers — is precisely what Kubernetes does not offer.

How a YAML List vs Map Difference Crash-Looped My Kyverno Fix

david — Mon, 12 Oct 2026 14:54:50 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Kyverno's cleanup-controller was restarting every few hours. The leader-election timeout was expiring under load — the same class of leader-election instability that CPU contention causes for etcd — the controller lost its lock, and the new leader took over, causing a brief gap in policy enforcement. The fix looked simple: increase leaderElectionRetryPeriod. The fix crash-looped the very pod it was meant to stabilize.

The Symptom

cleanup-controller-7f8b9c4d6-xk2p4   0/1     CrashLoopBackOff   3 (42s ago)

The pod started, immediately crashed, and Kubernetes restarted it. Three crashes in under two minutes. The Kyverno dashboard showed the cleanup controller as unavailable, and policy reports stopped updating.

The Investigation

The crash-loop meant the binary was rejecting its own flags. Checking the pod's logs:

Error: unknown flag: --0=--leaderElectionRetryPeriod=10s

The binary received --0=--leaderElectionRetryPeriod=10s as a single flag. That's not a valid argument — it's a rendered artifact of wrong YAML structure.

The Root Cause: List vs Map

The Helm chart's cleanupController.extraArgs expects a map, not a list:

# WRONG — list syntax, renders as --0=--leaderElectionRetryPeriod=10s
cleanupController:
  extraArgs:
    - "--leaderElectionRetryPeriod=10s"

# CORRECT — map syntax, renders as --leaderElectionRetryPeriod=10s
cleanupController:
  extraArgs:
    leaderElectionRetryPeriod: "10s"

Helm's range function iterates over maps as $key, $value pairs. When you pass a list, Helm treats each list item as a map entry with an integer index as the key. The rendered result:

# What the binary actually receives:
--0=--leaderElectionRetryPeriod=10s

# What it expected:
--leaderElectionRetryPeriod=10s

The binary sees --0 as a flag name, doesn't recognize it, and exits with a non-zero code. Kubernetes restarts the pod. Crash-loop.

Why This Is a Production Risk

This wasn't a staging environment. The Kyverno cleanup-controller handles:

Policy report cleanup — old reports pile up without it
Namespace selector caching — stale cache entries cause policy enforcement delays
Leader election — the restart cycle itself destabilizes the leader lock

The irony: the fix for leader-election instability caused a worse instability. The cleanup-controller restarted 3 times before I caught it — and during those restarts, policy reports accumulated without cleanup.

How to Catch This Before It Hits Production

1. Dry-run the Helm template:

helm template kyverno kyverno/kyverno -n kyverno \
  --set cleanupController.extraArgs.leaderElectionRetryPeriod=10s \
  | grep -A5 cleanup-controller

If the rendered YAML shows --0=... in the container args, the structure is wrong.

2. Check the chart's values.yaml:

helm show values kyverno/kyverno | grep -A10 extraArgs

The chart documents whether extraArgs is a map or list. Trust the chart, not your muscle memory from other Helm charts.

3. Use Kyverno's policy reports to validate:

After deploying, check the cleanup controller's pod status immediately:

kubectl get pods -n kyverno -l app.kubernetes.io/name=kyverno-cleanup-controller

If it shows CrashLoopBackOff within the first minute, the flags are wrong.

The Fix

# kubernetes/system/kyverno/application.yml
cleanupController:
  extraArgs:
    leaderElectionRetryPeriod: "10s"

One line change. The pod started cleanly, the crash-loop stopped, and policy reports resumed normal cleanup.

The Pattern

This is a common Helm gotcha across many charts, not just Kyverno:

Chart	Flag field	Type	Gotcha
Kyverno	`extraArgs`	map	List syntax renders `--0=` prefix
Cert-Manager	`extraArgs`	map	Same pattern
Prometheus	`extraArgs`	map	Same pattern
ArgoCD	`extraArgs`	map	Same pattern

The fix is always the same: if the chart expects a map, use map syntax. If you're unsure, helm template will show you exactly what the binary receives.

Further reading: For a deeper dive into distributed systems failure modes, Designing Data-Intensive Applications by Martin Kleppmann* explains why leader-election timeouts behave the way they do under I/O pressure.

Kyverno policies are only effective when the controller is running. A crash-looping cleanup controller means policy reports accumulate without cleanup, and namespace selector caches go stale. Pin your Helm values correctly, and dry-run before deploying to production.

Three Fixes Were Merged. None of Them Had Taken Effect.

david — Mon, 12 Oct 2026 14:54:14 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

GitOps makes a specific promise: the cluster matches git. When a change is merged, it deploys. When a resource is deleted from git, it's deleted from the cluster. Merge = deploy is the core guarantee the entire workflow rests on.

It's not always true. This is a story about finding three cases where it wasn't - two discovered during a drift audit, one that would have been inert even if ArgoCD had synced it correctly.

View the complete homelab infrastructure source on GitHub 🐙

How the Gap Opens: ArgoCD's App-of-Apps Bootstrap Problem

The cluster uses ArgoCD's app-of-apps pattern: a root Application watches a directory, creates child Applications for each manifest it finds, and those child Applications sync the actual workloads. The root Application uses a directory.include glob to decide which files to track.

The problem: that glob is set once, at bootstrap time, and files added to the directory later are only tracked if they match the existing glob. A file that doesn't match is invisible to ArgoCD - it won't create an Application for it, won't sync it, and won't alert about it. From ArgoCD's perspective, the file doesn't exist.

The monitoring stack had this problem. Two files - application.yml and manifests-application.yml - had been added to the kubernetes/system/monitoring/ directory but were missing from the parent Application's directory.include glob:

# monitoring-bootstrap/application.yml (the root)
spec:
  source:
    directory:
      include: "alertmanager-config.yml"  # ← only this file tracked
      # application.yml and manifests-application.yml: invisible

A fix for a broken configSecret reference had been merged to application.yml. It was valid YAML, it would have worked - but ArgoCD never picked it up. The live cluster was still running the broken version from initial bootstrap, indefinitely.

Finding the Gap: What the Drift Audit Actually Checks

The standard ArgoCD health check (kubectl get applications -n argocd) doesn't catch this. All Applications show Synced/Healthy. The bootstrapped-but-untracked files don't generate an Application at all, so there's nothing to show as out-of-sync.

The audit that found it was manual: compare the contents of each kubernetes/system/*/ directory against the tracked files in the corresponding bootstrap Application, and flag any file not in the glob:

# For each subdirectory, list tracked files vs actual files
for dir in kubernetes/system/*/; do
  echo "=== $dir ==="
  # What ArgoCD tracks (from the bootstrap Application's directory.include)
  # vs what's actually in the directory
  ls "$dir"*.yml 2>/dev/null
done

The audit found approximately 14 directories with the same untracked-after-bootstrap gap - no current drift in most of them, but nothing preventing drift from accumulating silently if files were added there in the future.

This is the same class of gap that a GitHub Actions Terraform-plan-to-Slack alert is designed to close on the Terraform side — a green pipeline or a Synced badge tells you the command ran, not that the running state matches intent.

The First Live Finding: Traefik allowCrossNamespace

Traefik's allowCrossNamespace fix had been merged to git on 2026-07-04. The change allowed IngressRoutes in one namespace to reference Services in another - without it, argo.woitzik.dev, monitoring.woitzik.dev, and status.woitzik.dev returned bare 404s.

The fix was in git. ArgoCD showed the Traefik Application as Synced. But the endpoints were still 404ing.

The cause: the Traefik Application's last successful sync was before the fix merged. ArgoCD's sync status reflects the last successful reconciliation, not a continuous comparison. If the sync was clean before the change and nothing triggered a re-sync after (no new commit to the watched path, no manual refresh), the Application can show Synced while being hours or days behind.

# Force ArgoCD to re-compare against current git HEAD
argocd app sync traefik --force
# or via annotation:
kubectl annotate application traefik -n argocd \
  argocd.argoproj.io/refresh=hard --overwrite

After manual sync, all three endpoints came back to 302 within seconds of the rollout completing.

The Second Finding: Velero's Fix That Couldn't Have Worked

Velero had a merged fix for repository maintenance frequency - too-frequent Kopia maintenance jobs were running during backup windows and causing timeouts. The fix set a custom maintenance frequency via Helm values.

The fix was synced correctly. It also did nothing, for two independent reasons:

Reason 1: Wrong Helm values structure.

The extraArgs key was placed at the top level of the Helm values instead of nested under configuration:

# What was merged (broken)
velero:
  extraArgs:
    - "--default-repo-maintain-frequency=168h"

# What Helm actually needs
velero:
  configuration:
    extraArgs:
      - "--default-repo-maintain-frequency=168h"

Helm silently drops unknown top-level keys. No error, no warning - the extra args simply weren't passed to the binary. Velero started with its defaults.

Reason 2: Wrong flag name.

The flag --default-repo-maintenance-frequency doesn't exist in Velero. The real flag is --default-repo-maintain-frequency (no "ance"). Even if the values structure had been correct, the binary would have printed an unknown flag error and ignored it.

Two independently-wrong things, each of which would have prevented the fix from taking effect on its own. Caught during the drift audit by checking the running Velero container's actual args:

kubectl get pod -n velero -l app.kubernetes.io/name=velero -o jsonpath=\
  '{.items[0].spec.containers[0].args}' | tr ',' '\n'
# → no --default-repo-maintain-frequency flag present

The correct fix, both issues resolved:

velero:
  configuration:
    extraArgs:
      - "--default-repo-maintain-frequency=168h"

The Three-Part Pattern

All three failures share the same shape:

	Traefik allowCrossNamespace	Velero maintenance frequency	Monitoring bootstrap files
Git state	Correct	Correct	Correct
Live state	Unsynced	Synced but inert	Never tracked
ArgoCD showed	`Synced/Healthy`	`Synced/Healthy`	No Application (invisible)
Found by	Manual sync + endpoint check	Checking running container args	Directory vs. glob comparison

In each case, the developer closed the PR with the reasonable belief that the fix was live. In each case, it wasn't. The merge was the easy part; the verification that the change actually took effect in the running cluster is the part that was skipped.

The Verification Step That Prevents It

For any change where "merged" isn't sufficient proof of "deployed," there's one check that works:

# For a Helm values change: verify the flag is in the running container
kubectl get pod -n <namespace> -l <label-selector> \
  -o jsonpath='{.items[0].spec.containers[0].args}'

# For a Kubernetes manifest change: verify the field is live
kubectl get <resource> -n <namespace> <name> \
  -o jsonpath='{.spec.<changed-field>}'

# For an ArgoCD Application bootstrap: verify the file is tracked
argocd app get <bootstrap-app> --output json | \
  jq '.spec.source.directory.include'

None of these are expensive. They're all one-liners. The cost of skipping them - in this case, Traefik returning 404 for three production subdomains, Velero running backup maintenance at the wrong frequency indefinitely, and monitoring fixes sitting inert - was substantially higher.

The same GitOps verification gap exists in Azure DevOps pipelines and GitHub Actions that deploy Terraform or Helm. A pipeline that reports "succeeded" proves that the deployment command ran without error - it doesn't prove that the cluster's running state matches the manifest. Checking the running state directly, after deployment, is the only check that closes the gap.

The Phoenix Project* is still the sharpest account I know of exactly this gap between "the change was made" and "the change took effect" - fiction, but painfully accurate about how it actually happens.

Postgres 16 18: Why You Can't Just Swap the Image

david — Mon, 05 Oct 2026 12:44:42 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

PostgreSQL major version upgrades are not swap-the-image operations. You can't change postgres:16 to postgres:18 in a Deployment and expect it to work. The data directory format changes, system catalogs are incompatible, and extensions need to be rebuilt.

When CNPG (CloudNativePG) upgraded from PG16 to PG18, the procedure required a fresh PVC, a full dump and restore, and a role discovery bug that nearly locked me out of the Authelia database.

View the complete homelab infrastructure source on GitHub 🐙

Why You Can't Swap the Image

PostgreSQL stores data in a format specific to its major version. The PG_VERSION file in the data directory tells PostgreSQL which version created it:

cat /var/lib/postgresql/data/PG_VERSION
# → 16

When PostgreSQL 18 starts and finds PG_VERSION = 16, it refuses to start:

FATAL: data directory has wrong ownership
HINT: The data directory was initialized by PostgreSQL 16. 
      Upgrade by running pg_upgrade.

pg_upgrade is the official tool for in-place major version upgrades. It copies data files from the old format to the new format, rewriting system catalogs and tuple headers. It works well on bare-metal PostgreSQL where you have direct filesystem access.

On Kubernetes with CNPG, pg_upgrade is not the right approach because:

CNPG manages the data directory through its operator — manual modifications are reverted
The PVC is bound to the cluster definition — changing the PostgreSQL version in the CRD doesn't automatically upgrade the data
CNPG's recommended migration path is dump-and-restore, not in-place upgrade

The Migration Procedure

Step 1: Dump from PG16

# Exec into the CNPG pod
kubectl exec -n database -it postgres-authelia-0 -- bash

# Dump the database
pg_dump -U postgres -Fc authelia > /tmp/authelia.dump

# Copy the dump to a temporary location
kubectl cp database/postgres-authelia-0:/tmp/authelia.dump ./authelia.dump

The -Fc flag produces a custom-format dump that's compressed and can be restored with pg_restore. The dump includes all data, schemas, roles, and extensions.

Step 2: Create Fresh PG18 Cluster

# kubernetes/system/postgres/cluster.yml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: postgres-authelia
  namespace: database
spec:
  imageName: ghcr.io/cloudnative-pg/postgresql:16.4  # temporary — will be updated
  instances: 1
  storage:
    size: 2Gi
    storageClass: local-path

Wait — the image is still PG16. That's intentional. CNPG creates the cluster with PG16 first, then upgrades the image to PG18 after the data is restored. This ensures the PVC and the operator agree on the initial state.

Step 3: Restore into PG18

After the cluster is running with PG16, update the image to PG18:

spec:
  imageName: ghcr.io/cloudnative-pg/postgresql:18.4

CNPG detects the image change, creates a new pod with PG18, and the old PG16 pod is terminated. The data directory is still PG16 format, so the new pod fails to start — which is expected.

Now restore the dump into the fresh PG18 data directory:

# Port-forward to the CNPG service
kubectl port-forward -n database svc/postgres-authelia 5432:5432 &

# Restore the dump
pg_restore -U postgres -d authelia --clean --if-exists ./authelia.dump

pg_restore handles the format conversion — it reads PG16-format data and writes it in PG18 format. The --clean --if-exists flags drop existing objects before restoring, ensuring a clean state.

Step 4: Update Authelia

Update the Authelia deployment to use the new Postgres 18 connection string (same host, same port, same database — the connection string doesn't change):

# Verify Authelia connects to the new database
kubectl logs -n apps -l app=authelia --tail=20
# → "Successfully connected to PostgreSQL 18.4"

The Role Discovery Bug

During the restore, pg_restore reported:

pg_restore: error: could not open input file "/tmp/authelia.dump": No such file or directory

The dump file was at a different path than expected. The real problem: the restore was running from a pod that had a different filesystem layout than the dump pod.

After fixing the path, the restore completed but Authelia failed to start:

FATAL: role "oc_dw" does not exist

The oc_dw role was a leftover from the original CNPG cluster initialization — CNPG creates a default operator role that isn't visible in pg_dump output because it's a replication role, not a regular database role.

The fix: create the missing role before restoring:

CREATE ROLE oc_dw WITH REPLICATION LOGIN;

The lesson: CNPG creates internal roles (oc_dw, streaming_replica) that aren't included in pg_dump output. After a dump-and-restore migration, these roles must be recreated manually.

The Fresh PVC Requirement

The critical step that most tutorials skip: you need a fresh PVC. You can't restore a PG16 dump into a PG16 data directory and then expect PG18 to read it. The data directory must be empty — PG18 creates its own data directory format on first start, and pg_restore populates it.

In CNPG, this means:

Delete the existing PVC (data loss — you need the dump)
Let CNPG create a new PVC with the PG18 image
Restore the dump into the fresh cluster

The PVC deletion is the scary part. If the dump is corrupted or incomplete, the data is gone. The verification before deletion:

# Verify dump is complete
pg_restore --list ./authelia.dump | wc -l
# → Should show hundreds of objects (tables, sequences, functions)

# Verify dump integrity
pg_restore --verbose --no-owner --no-privileges --dry-run ./authelia.dump
# → Should complete without errors

What I'd Change

Use CNPG's Backup/Restore instead of manual dump. CNPG supports pg_basebackup and WAL archiving to S3. A CNPG backup includes the operator roles and can be restored directly without the oc_dw gotcha. The manual dump approach was chosen because the existing cluster wasn't configured for CNPG backups at the time of migration.
Test the migration on a non-production cluster first. The oc_dw role discovery happened during the Authelia migration — the only SSO for every service. If the restore had failed, every service would be unreachable.

PostgreSQL major version upgrades on Kubernetes are the same challenge as Azure Database for PostgreSQL Flexible Server upgrades: Azure handles the in-place upgrade automatically, but the same data directory format incompatibility exists. The difference is that Azure abstracts the dump-and-restore behind a API call, while Kubernetes requires you to do it manually. The underlying PostgreSQL constraint is identical: major versions are not backward-compatible at the storage layer.

The Linux Command Line* is worth having on the shelf for exactly this kind of migration - pg_dump, pg_restore, kubectl cp, and a dozen other shell tools chained together under time pressure go a lot smoother when the shell itself isn't also something you're learning in the moment.

Renovate in Kubernetes: OOM, Hashicorp Downloads, and the 6GB Container

david — Sun, 04 Oct 2026 11:30:39 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Renovate is the dependency update bot that keeps my k3s cluster current, and it's the "auto-update" half of the operating model that decides what gets merged automatically vs. what waits for a human. It runs as a Kubernetes CronJob every 2 hours, checks for new container images, Helm chart versions, and Terraform provider releases, and opens PRs on GitHub.

For two weeks, it was failing silently on every run. The CronJob reported "Completed." The GitHub commits showed no new PRs. The logs showed nothing — because Renovate exits cleanly on config validation errors, and the CronJob's successfulJobsHistoryLimit kept only the last successful run.

View the complete homelab infrastructure source on GitHub 🐙

Failure 1: Invalid Preset

The first failure was a config error. The renovate.json file referenced an invalid preset:

{
  "extends": ["config:base", ":enableHelpfulPre-commit"]
}

:enableHelpfulPre-commit is not a real Renovate preset. Renovate's config validation caught the error and exited with a zero exit code — because config validation errors are treated as "nothing to do," not "something broke."

The CronJob's exit code was 0. Kubernetes considered the job successful. No alert fired. Renovate did nothing on every run.

The fix: remove the invalid preset and replace the deprecated config:base with config:recommended.

{
  "extends": ["config:recommended"]
}

Failure 2: OOM at 512Mi

After fixing the config, Renovate started running — and immediately OOM'd.

The container limit was 512Mi. Renovate's Node.js runtime, plus the GitHub API client, plus the Terraform provider registry client, plus the container image metadata parser, consumed more than 512Mi during a full scan.

The symptom: the Pod restarted with OOMKilled status. Kubernetes restarted it. The next run OOM'd again. After 3 failures, the CronJob marked the job as failed — but failedJobsHistoryLimit: 3 meant only the last 3 failures were visible, and they were being garbage-collected faster than I checked.

kubectl get jobs -n apps -l app=renovate
# NAME              COMPLETIONS   DURATION   AGE
# renovate-28197    0/1           ...        2m
# renovate-28196    0/1           ...        2h
# renovate-28195    0/1           ...        4h

The fix: raise the memory limit to 1Gi and set NODE_OPTIONS to limit the V8 heap:

env:
  - name: NODE_OPTIONS
    value: "--max-old-space-size=896"
resources:
  limits:
    memory: 6Gi
  requests:
    memory: 1Gi

The NODE_OPTIONS limit of 896MB (leaving ~100MB for native code and runtime overhead) prevents V8 from consuming the entire container memory during garbage collection cycles. Without it, V8 allocates memory up to the container limit and gets OOMKilled before the next GC cycle.

Failure 3: Terraform Provider Downloads

The third failure was the most subtle. Renovate was consuming 6GB of memory during Terraform provider update checks. The reason: Renovate downloads Terraform provider binaries to verify version compatibility.

For each Terraform provider in the repo (routeros, proxmox, cloudflare, garage), Renovate:

Queries the Terraform Registry API for available versions
Downloads the provider binary for the current and latest versions
Compares binary compatibility (provider schema, API version)
Opens a PR if a newer version is available

Step 2 is the memory problem. Terraform provider binaries are 50-200MB compressed. With 4 providers, plus their dependencies, Renovate downloads ~500MB of provider binaries during a full scan. The decompressed binaries and their metadata consume 2-3x that in memory.

The fix was two-part:

Raise the memory limit to 6Gi — enough for the full scan including provider downloads
Limit concurrent requests to the Terraform Registry:

{
  "terraform": {
    "concurrentRequestLimit": 2
  }
}

With concurrentRequestLimit: 2, Renovate downloads at most 2 provider binaries simultaneously, reducing peak memory usage from ~6GB to ~3GB. The scan takes longer, but it completes within the 6Gi limit.

The Silent Failure Pattern

The common thread across all three failures: Renovate exited cleanly. The CronJob reported success. No alert fired. The only evidence of failure was the absence of new PRs on GitHub.

This is the silent failure pattern:

The application fails internally but exits with code 0
Kubernetes considers the job successful
The successfulJobsHistoryLimit preserves the "successful" job
Nobody checks because the job "succeeded"

The fix: add a post-run check that verifies Renovate actually did something:

# Post-run check: verify PRs were opened
- name: Verify Renovate activity
  script: |
    PR_COUNT=$(gh pr list --repo dwoitzik/homelab-infrastructure \
      --author "app/renovate" --state open --json number --jq 'length')
    if [ "$PR_COUNT" -eq 0 ] && [ "$(date +%u)" -le 5 ]; then
      echo "WARNING: No open Renovate PRs on a weekday"
      # Alert via Discord webhook
    fi

The Current Configuration

# kubernetes/apps/renovate/renovate.yml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: renovate
  namespace: apps
spec:
  schedule: "0 */2 * * *"  # every 2 hours
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: renovate
              image: ghcr.io/renovatebot/renovate:43.245.0
              env:
                - name: RENOVATE_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: renovate-token
                      key: github-pat
                - name: NODE_OPTIONS
                  value: "--max-old-space-size=896"
              resources:
                limits:
                  memory: 6Gi
                  cpu: 2000m
                requests:
                  memory: 1Gi
                  cpu: 100m
          restartPolicy: Never
      backoffLimit: 2

The restartPolicy: Never with backoffLimit: 2 means the CronJob retries twice on failure, then gives up. Combined with the memory limit and heap size cap, Renovate completes its scan within resource bounds.

The Lesson

Silent failures in CronJobs are the most dangerous failure mode in a Kubernetes cluster — the same "reports success but did nothing real" pattern as ArgoCD showing Synced/Healthy on an Application that never actually applied. The job "succeeds," the operator doesn't check, and the dependency update pipeline silently stops working. Weeks pass without updates, security patches don't apply, and nobody notices until a vulnerability is disclosed for a package that Renovate would have updated.

The fix isn't just more memory — it's observability on the pipeline itself. Every CronJob that performs a critical function should have a post-run verification that confirms it actually did its job, not just that it exited cleanly.

CronJob observability is the same problem in Azure DevOps: a pipeline that succeeds but produces no artifact is indistinguishable from a pipeline that didn't run. Azure Monitor's Pipeline Analytics tracks success rate, not "did the pipeline produce meaningful output." The fix in both cases is the same: add a verification step that checks for the expected outcome (PRs opened, artifacts published, deployments completed) and alerts on absence.

The Phoenix Project* tells the same story from the human side of this exact failure mode - the automated system that everyone assumes is working, until someone finally checks.

Longhorn to NFS: Why Distributed Storage Didn't Make Sense Here

david — Mon, 28 Sep 2026 13:44:58 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Longhorn is designed for multi-node Kubernetes clusters. It replicates PersistentVolume data across nodes, so if one node dies, the data survives on the other two. It's a great distributed storage system.

I run three k3s nodes on a single physical host. All three nodes share the same NVMe. When Longhorn replicates a PVC across three nodes, all three replicas live on the same physical disk. There is no distribution. There is no redundancy. And when a node failed, Longhorn's own high-availability logic created a problem that wouldn't exist without it.

View the complete homelab infrastructure source on GitHub 🐙

The Multi-Attach Error

When a k3s node becomes unreachable (OomKill, kubelet crash, network blip), Longhorn marks its replicas as "rebuilding" and tries to attach the volume to a healthy node. But Longhorn's RWO (ReadWriteOnce) volumes can only be attached to one node at a time.

The sequence:

k3s-12 becomes unreachable (kubelet lost lease)
Longhorn detaches the PVC from k3s-12
Longhorn tries to attach the PVC to k3s-13
k3s-12 recovers and comes back online
Longhorn sees both nodes claiming the volume
Multi-Attach error: the volume is "attached" to two nodes simultaneously

The pod that needs the volume (Postgres, for example) can't start on either node because Longhorn refuses to mount a volume that's in a Multi-Attach state. The fix requires manually detaching the volume from both nodes and letting Longhorn re-attach it cleanly — a manual step during an outage, exactly when automation should be working.

On a multi-host cluster, this is a real HA scenario — the volume legitimately needs to failover to a different physical disk. On a single-host cluster, the "failover" is to the same NVMe, and the Multi-Attach error is pure overhead.

The NFS Alternative

NFS (Network File System) doesn't have a Multi-Attach problem because it's not a block storage system. NFS exports a directory over the network. Any number of clients can mount it simultaneously. There's no "attachment" state to conflict.

The NFS server runs on a dedicated LXC (ct-srv-nfs-01) with a ZFS-backed dataset:

# terraform/stacks/proxmox/lxc.tf
resource "proxmox_virtual_machine" "ct_srv_nfs_01" {
  vm_id   = 220
  name    = "ct-srv-nfs-01"
  memory { dedicated = 2048 }
  # ...
}

The k3s NFS provisioner creates PVCs on this NFS server:

# kubernetes/system/nfs-provisioner/application.yml
apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    helm:
      values:
        nfs:
          server: 10.0.20.100
          path: /archive
        storageClass:
          name: nfs-client
          defaultClass: true

When k3s-12 goes down, the pods reschedule to k3s-11 or k3s-13, mount the same NFS export, and continue where they left off. No Multi-Attach error, no manual detachment, no volume attachment state machine. NFS handles concurrent mounts natively.

The Trade-offs

What NFS Gives Up

Feature	Longhorn	NFS
Replication across nodes	Yes (3x by default)	No (single server)
Snapshots	Yes (per-volume)	Yes (ZFS snapshots)
ReadWriteMany	Yes	Yes
CSI driver	Yes	Yes (nfs-subdir-external-provisioner)
Performance	Block-level (faster)	Network-mounted (slower)
Node failure tolerance	Yes	No (single server)

NFS gives up replication and node-failure tolerance. If the NFS server dies, every PVC it serves is unavailable. On a single physical host, this is the same risk as Longhorn — both storage systems live on the same NVMe. Longhorn's replication doesn't help when the underlying disk is the single point of failure.

What NFS Gives Back

No Multi-Attach errors. Pods reschedule freely without volume attachment conflicts.
Simpler failure modes. NFS is either up or down. No "rebuilding," "degraded," or "reverted" states.
ZFS snapshots for backup. PBS backs up the NFS server's ZFS dataset, giving point-in-time recovery without Longhorn's snapshot overhead.
Lower resource usage. Longhorn runs a per-node process (manager + driver) consuming ~200MB RAM per node. NFS is a single process on one LXC.

The Storage Decision Matrix

After migrating from Longhorn to NFS, I defined the rule:

Application	Storage Class	Reason
PostgreSQL (CNPG)	local-path	CNPG manages its own replication
Garage S3 metadata	local-path	SQLite file-locking (the NFS trap)
Mealie, Home Assistant	local-path	SQLite databases
Everything else	nfs-client	Default, survives pod rescheduling

The principle: embedded databases go on local-path (file-locking), everything else goes on NFS (survivability). PostgreSQL doesn't fit either category because CNPG manages its own PV independently.

The Migration

Moving PVCs from Longhorn to NFS required:

Deploy the NFS server LXC with the correct ZFS dataset
Install the nfs-subdir-external-provisioner Helm chart
Copy data from Longhorn volumes to NFS (using kubectl cp or a temporary pod)
Update each application's storageClassName from longhorn to nfs-client
Delete the old Longhorn PVC (data is already on NFS)
Remove Longhorn from the cluster

Step 5 is the nerve-wracking part — deleting a PVC that contains production data. The verification before deletion:

# Verify data exists on NFS
kubectl exec -it temp-pod -- ls -la /mnt/nfs/authelia-data/
# → Confirm database files, config, etc.

# Verify application starts with NFS PVC
kubectl delete pod authelia-xxxxx  # force reschedule
# → Pod starts, mounts NFS, passes health checks

After the migration, Longhorn was uninstalled. The cluster went from three storage replicas on one disk to a single NFS server on the same disk — simpler, more predictable, and without the Multi-Attach false alarm.

Distributed storage on a single host is the same anti-pattern as Azure Zone-Redundant Storage across availability zones that share the same power source. If the underlying infrastructure isn't actually distributed, the replication layer adds complexity without adding resilience. The fix in both cases: match the storage topology to the actual infrastructure topology. Single host = single NFS server. Multi-host = distributed storage.

Designing Data-Intensive Applications* has the clearest explanation I've read of why replication only buys you resilience when the replicas are actually independent - which is the whole lesson of this migration in one sentence.

The Operating Model: What Should Auto-Update vs. What Needs a Human

david — Fri, 25 Sep 2026 11:00:27 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

A homelab that requires daily attention isn't a homelab — it's a job. The goal: a cluster that runs for weeks without intervention, alerts when something breaks, self-heals what it can, and waits for a human only when a human is actually needed.

This is the operating model that makes that work. Not aspirational — documented from what actually runs in production, including the deliberate gaps and the non-goals that keep the system honest.

View the complete homelab infrastructure source on GitHub 🐙

What Auto-Updates

Renovate runs every 2 hours via CronJob, tracking container images, Helm charts, and Terraform providers. But not everything gets the same treatment:

Auto-Merge (Patch/Minor/Digest, 3-Day Soak)

Stateless workloads that can be rolled back by ArgoCD's selfHeal if they break:

Dashboard widgets (Homepage, Uptime Kuma display)
Development tools (pre-commit hooks, linters)
Non-critical utilities (SearXNG, Mealie, MySpeed)

Renovate opens a PR, CI runs, and after 3 days of green the PR auto-merges. If the new version breaks, ArgoCD's syncPolicy.automated.selfHeal: true reverts to the previous version on the next sync.

PR-Only, Always (Manual Review Required)

Stateful and critical services where a bad upgrade can cause data loss or auth outage:

Databases: CNPG Postgres, Redis/Valkey
Auth: Authelia, Vault
Storage: Garage S3, Velero
Apps with data migrations: Nextcloud, Paperless, Gitea, Immich

Every update — patch, minor, or digest — gets a PR that requires manual review. The failure mode for these isn't "pod restarts" — it's "database schema mismatch" or "OIDC keys rotated."

PR-Only, Always (Major Version Bumps)

Every major version bump, any package, regardless of tier. Major versions break APIs, change configuration formats, and introduce migration steps that no automated tool can reliably handle.

Terraform: Never Auto-Applied

Every Terraform change goes through Atlantis: PR → plan → atlantis apply comment → apply. No exceptions. Terraform changes physical and VM state on a single-point-of-failure host — that always gets a human in the loop.

What Alerts (and Where)

Prometheus + Alertmanager route to Discord via webhook. ArgoCD's notifications-controller sends app-state events through a separate path.

Critical Alerts (Always Discord)

Everything with severity: critical:

Proxmox host temperature
k3s control-plane down
Storage >93% full
Postgres replication broken
Certificate expiry soon
Velero backup failed

Warning Alerts (Selective)

Only explicitly named warnings that matter at steady state:

KubePodCrashLooping — a pod is crash-looping
KubeJobFailed — a CronJob failed (covers Renovate itself)
KubeNodeNotReady — a node is unreachable
VeleroBackupPartialFailure — backup completed with warnings

Deliberately NOT Alerted

Routine warnings that would make Discord noisy without being actionable:

KubeMemoryQuotaOvercommit — LXC soft ceilings, not real pressure
KubeCPUThrottling — normal behavior for bursty workloads
KubePodNotReady during rolling updates — temporary, self-resolving

The principle: an alert nobody acts on trains people to ignore Discord. Every alert must have a corresponding action in the runbook. If there's no action, there's no alert.

What Self-Heals

ArgoCD: Automatic Sync + Self-Heal

All 41 live Applications have syncPolicy.automated.selfHeal: true. A merged manifest change deploys itself. Manual kubectl drift on anything ArgoCD tracks gets reverted automatically within minutes.

This is the primary self-healing mechanism. If someone manually patches a Deployment (during debugging, for example), ArgoCD reverts it on the next sync cycle. The cluster always converges to the git state.

Docker Containers: restart: unless-stopped

Media stack, Minecraft, AdGuard, Unbound — all run with restart: unless-stopped in Docker Compose. Crashes and host reboots self-recover the process. The data recovery is handled by NFS/ZFS, not Docker.

Kyverno: Audit-Only (Deliberately)

Kyverno runs in Audit mode — it reports policy violations in PolicyReports but does not block or auto-remediate. This is deliberate:

A bad policy in Enforce mode silently blocks legitimate deploys
On a single-operator homelab, there's nobody to notice the block quickly
Audit mode provides the visibility without the blast radius

The policies exist (require-resource-limits, disallow-latest-tag, disallow-privileged-containers) but they're watching, not enforcing. When I'm confident they won't cause false positives, they'll flip to Enforce.

What Stays Manual (On Purpose)

Terraform Apply

Always through Atlantis. A human comments atlantis apply. Never auto-applied, regardless of what changed. This is the one rule with no exceptions.

Stateful Service Bumps

Authelia, Vault, CNPG, Garage — reviewed before merge. The failure mode is data loss or auth outage, not "reroll the pod."

Snapshots Before State-Affecting Changes

Proxmox VM/CT snapshot or Velero PV snapshot, taken manually before applying anything that touches running state. The judgment of "is this change state-affecting" doesn't belong to a script.

Velero Restore Testing

Running, tested successfully at least once per namespace. But not yet exercised across every service. A backup that's only partly restore-tested is closer to hope than guarantee.

Full Cluster Rebuild

Documented in DISASTER-RECOVERY.md. Not automated. The target is fast, well-documented recovery — not zero-touch failover, because true HA isn't achievable on one physical host.

The Non-Goals

These are explicitly not targets:

Zero-downtime HA. Not achievable with one physical host. Not attempted.
Fully unattended major-version upgrades. Stateful service upgrades have repeatedly needed human judgment mid-migration (Postgres mount-point gotcha, capability-drop regression). Automating past that trades a 10-minute manual step for a much longer unattended-failure cleanup.
Alerting on everything. Deliberately tuned to steady-state-relevant signals only.

The Contract

This document is the contract between the infrastructure and the operator. Everything not listed here is assumed to be working. If it's not working and it's not in this document, it's a gap — not an expected manual task.

The operating model is a living document. As the cluster evolves (Cilium CNI, R2 offsite backup, Kyverno enforcement), the categories shift. But the principle stays: automate what's safe, alert what's important, and leave everything else to a human with context.

This operating model maps directly to enterprise SRE practices: SLO-based alerting replaces threshold alerting, runbooks define the human response for each alert type, and change management gates (like Atlantis's apply requirement) prevent unreviewed infrastructure changes. The difference is that enterprise environments have teams; a homelab has one person who needs to sleep through the night without Discord notifications.

The Phoenix Project* is the book that most shaped this document - it's fiction, but the underlying argument (know exactly what's automated, what's gated, and why) is the same one this operating model tries to make explicit instead of leaving implicit.

CPU Scheduling for etcd: Why Proxmox cpu.units Matters

david — Mon, 21 Sep 2026 12:35:09 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Update: this cluster has since moved off etcd entirely — a single-server k3s setup uses the
embedded SQLite datastore by default, and the multi-server HA path that would have used etcd
was never turned on here. The CPU-contention problem and the cpu.units fix below were real
while etcd was in play, and the underlying lesson (latency-sensitive, write-heavy workloads need
scheduling priority when they share a host with bursty ones like LLM inference or game servers)
still holds for anyone running real multi-server etcd today. Left as a historical/general
reference rather than a description of this cluster's current architecture.

etcd is the brain of Kubernetes. Every API call, every configmap update, every pod scheduling decision goes through etcd. When etcd is slow, everything is slow. When etcd times out, the API server becomes unreachable.

On a single Proxmox host running k3s VMs alongside Ollama LLM inference, Minecraft game servers, and Docker media workloads, etcd shares CPU with everything else. Without scheduling priority, a Minecraft player join and an Ollama model load can delay etcd's fdatasync calls enough to trigger leader-election timeouts.

The fix: cpu.units = 2048 on k3s VMs, giving them 2x the CPU scheduling priority over every other workload on the host — the same VMs that get staggered boot ordering to prevent I/O storms in the first place.

View the complete homelab infrastructure source on GitHub 🐙

How Proxmox CPU Scheduling Works

Proxmox uses the Linux CFS (Completely Fair Scheduler) with added weight controls. Each VM gets a cpu.units value that determines its share of CPU time when multiple VMs compete for the same physical cores.

The default is units = 1024. When two VMs with equal units compete for a single core, they each get 50% of the CPU time. When one VM has units = 2048 and another has units = 1024, the first gets 2/3 and the second gets 1/3.

This is not CPU pinning (which restricts a VM to specific cores). It's CPU weighting (which determines priority when cores are shared). On a host with 16 threads and 12+ VMs/LXCs, most cores are shared between multiple workloads.

The etcd Problem

etcd's performance depends on write latency. Every key-value operation (lease renewal, configmap update, secret sync) requires an fdatasync to the WAL (Write-Ahead Log) on disk. etcd's leader-election timeout is 5 seconds — if the leader can't renew its lease within that window, controller-runtime terminates the process.

On a single NVMe shared by all VMs, etcd's fdatasync competes with:

Ollama reading model weights from disk (14GB qwen2.5 model)
Minecraft world writes from the DMZ game server
NFS serving PVC data to k3s pods
ZFS txg commits flushing dirty data

Under normal load, this is fine — NVMe IOPS are high enough to handle all of them. But when Ollama starts loading a 26B model (Gemma 4) and Minecraft generates terrain simultaneously, the NVMe queue depth spikes, fdatasync latency increases, and etcd's lease renewal window shrinks.

Without CPU priority, etcd's fdatasync also competes for CPU time with these workloads. The kernel's CFS scheduler doesn't know that etcd's fdatasync is more important than Ollama's matrix multiplication — it just sees two processes requesting CPU time and allocates it equally.

The Fix

# terraform/stacks/proxmox/vm.tf

# k3s VMs — 2x scheduling priority
resource "proxmox_virtual_machine" "vm_srv_k3s_11" {
  cpu {
    cores = 4
    units = 2048  # 2x default
  }
}

# Ollama LXC — default priority
resource "proxmox_virtual_machine" "ct_srv_ai_01" {
  cpu {
    cores = 6
    units = 1024  # default
  }
}

# Minecraft LXC — default priority
resource "proxmox_virtual_machine" "ct_dmz_games_01" {
  cpu {
    cores = 2
    units = 1024  # default
  }
}

The k3s VMs get units = 2048. Everything else stays at the default 1024. When etcd and Ollama compete for the same CPU cycle, etcd gets 2/3 of the time and Ollama gets 1/3.

This doesn't reduce Ollama's throughput under normal conditions — when there's no contention, Ollama still gets 100% of the CPU it requests. The priority only kicks in when multiple workloads compete for the same cores simultaneously.

The Evidence

Before cpu.units = 2048, the CNPG operator was restarting due to leader-election timeouts:

kubectl get pods -n cnpg-system
# NAME                                    RESTARTS
# cnpg-cloudnative-pg-5f8b9c4d6-xk2p4   310

310 restarts in 21 days. Each restart's log showed:

Leader election retry deadline exceeded
context deadline exceeded

After applying cpu.units = 2048 and the corresponding leader-election timeout increase (--leader-renew-deadline=50), restarts dropped to zero.

The CPU priority alone didn't fix it — the leader-election timeout increase was also necessary. But the CPU priority reduced the frequency of fdatasync delays enough that the 50-second deadline is never challenged under normal load.

The RAM Priority Interaction

CPU priority and memory priority are separate in Proxmox. cpu.units affects CPU scheduling; memory.dedicated and memory.floating affect RAM allocation.

On this host, both matter:

etcd needs low-latency CPU for fdatasync (solved by cpu.units)
k3s control-plane needs guaranteed RAM for API server and scheduler (solved by memory.dedicated = 12284, the same VM-vs-LXC memory model covered in the overcommit guard writeup)

The RAM allocation is a hard reservation — 12 GB is always available for the k3s-11 VM. The CPU priority is a soft weighting — it only matters when cores are shared. Both are necessary: without RAM priority, the k3s VM could be ballooned down to 4 GB under pressure; without CPU priority, etcd could lose the fdatasync race to Ollama.

What I'd Change

Pin etcd to specific cores. CPU pinning would guarantee etcd always has CPU available, rather than just having priority. But pinning reduces overall CPU utilization — a pinned core can't be used by other VMs even when etcd is idle. On a 16-thread host with 12+ workloads, the utilization loss isn't worth it.
Monitor etcd fdatasync latency directly. Prometheus can scrape etcd's etcd_disk_wal_fsync_duration_seconds metric. An alert on p99 > 100ms would catch CPU contention before it triggers leader-election timeouts.

CPU scheduling priority is the same concept as Azure VM series selection: E-series VMs are memory-optimized, F-series are compute-optimized, and Dv5-series offer balanced resources. Choosing the wrong series for etcd (a latency-sensitive, write-heavy workload) produces the same performance degradation as running it without cpu.units on Proxmox. The difference is that Azure makes the choice at VM creation time, while Proxmox lets you adjust it dynamically.

Designing Data-Intensive Applications* is the best resource I know for understanding why a consensus system like etcd is so much more latency-sensitive than an ordinary stateless workload in the first place.

3-2-1 Backup in Practice: Velero + PBS + Offsite That Doesn't Work Yet

david — Mon, 21 Sep 2026 12:24:26 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

The 3-2-1 backup rule is simple: 3 copies of your data, on 2 different media types, with 1 copy offsite. My homelab implements all three layers. In theory. In practice, each layer has its own failure mode that I discovered only when I needed it — starting with the Garage S3 target Velero backs into, which is where layer 1 lives.

View the complete homelab infrastructure source on GitHub 🐙

The Three Layers

Layer 1: Velero → Garage S3 (in-cluster, daily 05:00 UTC)
    ↓
Layer 2: PBS → External HDD (local, daily 03:00 UTC)
    ↓
Layer 3: rclone → Google Drive (offsite, DISABLED)

Layer 1: Velero → Garage S3

Velero backs up k3s namespaces (apps, vault, database, argocd) to Garage S3 — a self-hosted S3-compatible object store running inside the same cluster.

# kubernetes/system/velero/schedule.yml
spec:
  schedule: "0 5 * * *"
  template:
    defaultVolumesToFsBackup: true
    includedNamespaces: ["apps", "vault", "database", "argocd"]
    excludedResources: ["events", "events.events.k8s.io"]
    ttl: 720h  # 30 days

defaultVolumesToFsBackup: true was the critical fix. Without it, Velero only captured Kubernetes manifests — not the actual PVC data. The backups "completed" for weeks with zero data.

After the fix, Kopia sidecars run alongside each pod, copying PVC contents to the Garage S3 velero bucket. Daily backups, 30-day retention.

The catch: Garage runs inside the cluster. If the cluster dies, both Velero and Garage are gone. Layer 1 is useful for recovering individual PVCs or namespaces within a running cluster — not for full cluster recovery.

Layer 2: PBS → External HDD

Proxmox Backup Server (PBS) backs up VM and LXC disk images to a Seagate 2TB External HDD* connected via USB.

# PBS backup job — runs daily at 03:00 UTC
# Backs up all VMs and LXCs to /mnt/backup (USB HDD)
proxmox-backup-client backup \
  --repository local:/mnt/backup \
  --ns homelab \
  vm/110/pct/200/pct/210/...

PBS handles deduplication, compression, and incremental backups. The USB HDD provides local, offline backup that survives cluster failures.

The gotcha: PBS and the k3s VMs share the same physical host. A host-level failure (PSU, NVMe death) takes out both the primary data and the PBS backup. Layer 2 is protection against software failure (corruption, accidental deletion), not hardware failure.

Layer 3: rclone → Google Drive (DISABLED)

The offsite layer was supposed to be rclone syncing from Garage S3 to Google Drive. In practice, Google's API throttled the sync to 1.6 KiB/s:

Transferred:   1.6 KiB / 4.2 GiB,  0.00%
Elapsed time:  2h 30m
Transfer rate: 1.6 KiB/s

Google Drive's API rate limiting for server-to-server transfers (no user interaction) is aggressive. For 4 GB of data, the sync would take days. Combined with Google's 750 GB/day upload limit for personal accounts, and the fact that the sync would need to run continuously to keep up with daily backups, the offsite layer was disabled.

The replacement (Cloudflare R2) is scaffolded but not active:

# kubernetes/system/velero/offsite-schedule.yml
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily-offsite
spec:
  schedule: "0 4 * * *"
  template:
    includedNamespaces: ["apps", "vault", "database", "argocd"]
    storageLocation: r2-offsite
    ttl: 168h  # 7 days

Waiting on a Cloudflare R2 account with real API credentials. R2 has no egress fees and no API rate limiting for the volume this backup produces (~4 GB/day). Activating it also fixes a second problem beyond throttling: Garage runs inside the same cluster Velero is protecting, so an in-cluster-only backup target is circular regardless of how fast it uploads.

The Restore Gotcha

The first time I tested a full restore, PBS hit an IP/MAC conflict:

Error: VM 211 is running on a different node (10.0.20.11)
Proxmox cannot start the VM because the MAC address is already in use

The issue: PBS restores the VM's network configuration exactly as it was, including the MAC address. If the original VM is still running (or its MAC is cached in the bridge), the restored VM can't start because Proxmox detects a MAC conflict on the virtual bridge.

The fix: restore to a temporary VMID, verify the restore is complete, then shut down the original and rename the restored VM. This is a manual multi-step process — not something you want to do under time pressure during a real disaster.

What Actually Works

Layer	Protects Against	Doesn't Protect Against
Velero → Garage	PVC corruption, namespace deletion, accidental kubectl delete	Cluster-wide failure (Garage is in-cluster)
PBS → USB HDD	Software corruption, accidental VM deletion, ZFS pool issues	Host hardware failure (PBS is on the same host)
rclone → Google Drive	Host hardware failure, theft, fire	Nothing yet — disabled due to throttling

The honest assessment: I have two functional backup layers, both on the same physical host. True offsite backup doesn't exist yet. The 3-2-1 rule is aspirational, not achieved.

The Verification Gap

The most important lesson: a backup you haven't restored is not a backup.

Velero backups run daily, PBS runs daily, and until recently neither had been fully restored. The Velero defaultVolumesToFsBackup gap existed for weeks because nobody tested a restore. The PBS IP conflict was discovered during the first full restore test.

After these discoveries, I added:

Monthly Velero restore test — restore the database namespace to a temporary namespace, verify Postgres starts and contains expected data
Quarterly PBS restore test — restore one VM to a temporary VMID, verify it boots and services are functional
Post-backup verification — velero backup describe --details | grep "Pod Volume Backups" after every scheduled backup

Backup verification is the same compliance requirement in Azure: ISO 27001 and NIS2 both require documented, tested restore procedures. Azure Backup reports "Completed" for VM snapshots, but the snapshot might not include the data disk if the backup policy was misconfigured. The only way to verify is to actually restore a test VM and confirm its contents — the same monthly drill I now run against Velero and PBS.

NFS vs local-path: The SQLite Trap That Corrupted My S3 Metadata

david — Mon, 14 Sep 2026 12:43:37 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Garage S3, my self-hosted S3-compatible object store, uses SQLite for its metadata database. One morning, Terraform state operations started failing with SQLITE_CORRUPT. The bucket metadata was gone. The terraform-state bucket, the Atlantis lock table, the Velero backup index — all of it stored in a SQLite file that was now corrupted.

The root cause: Garage was running on an NFS-backed PersistentVolume. NFS doesn't support the file-locking primitives that SQLite's WAL (Write-Ahead Logging) mode requires. Under concurrent access — Garage's metadata writer and a Velero backup reading the same database — the NFS lock delegation failed silently, and SQLite wrote to overlapping pages.

View the complete homelab infrastructure source on GitHub 🐙

The Storage Classes

My k3s cluster has two StorageClasses:

# NFS — for most workloads
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-client
provisioner: nfs-subdir-external-provisioner
parameters:
  server: 10.0.20.100
  path: /archive
reclaimPolicy: Retain

# Local-path — for embedded databases
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-path
provisioner: rancher.io/local-path
reclaimPolicy: Retain

NFS (nfs-client) is the default. It's backed by a dedicated LXC running an NFS server on ZFS, providing storage that survives pod rescheduling — a pod on k3s-12 can access the same PVC as a pod on k3s-13 because the NFS server is independent of any specific node.

Local-path (local-path) pins the PV to whichever node created it. If the pod reschedules to a different node, the PVC is inaccessible until the pod returns to the original node. This is a limitation, but it's the right trade-off for certain workloads.

Why SQLite and NFS Don't Work

SQLite's WAL mode requires fcntl() file locks — specifically, F_SETLK (non-blocking lock) and F_SETLKW (blocking lock). These locks coordinate access between concurrent processes writing to the same database file.

NFS handles file locks differently:

NFSv3: No native lock support. fcntl() calls return success but locks are local to the client — two NFS clients can both acquire an "exclusive" lock on the same file simultaneously.
NFSv4: Has LOCK operations, but the lock delegation model introduces latency and failure modes that SQLite's tight locking loop doesn't tolerate. If the NFS server is slow to respond to a lock request, SQLite's default 5-second busy timeout can expire, causing the application to retry — and the retry can conflict with the lock held by another client.
Kubernetes NFS provisioner: The nfs-subdir-external-provisioner uses NFSv4, but the lock delegation is handled by the NFS server's rpc.lockd daemon, which runs in a separate process space. Under concurrent load, lockd can lose track of which client holds which lock.

The result: SQLite thinks it has an exclusive lock, but another process (or the same process on a different connection) also has a lock. Both write to the database file. Pages overlap. The database corrupts.

The Garage Incident

Garage runs with two storage mounts:

# kubernetes/apps/garage/garage.yml
volumes:
  - name: data
    persistentVolumeClaim:
      claimName: garage-data      # NFS — bucket objects
  - name: meta
    persistentVolumeClaim:
      claimName: garage-meta      # local-path — SQLite metadata

The data volume (bucket objects) is on NFS — fine, because S3 object storage doesn't use file locks for individual files. The meta volume (SQLite database) was also on NFS initially. This worked until a Velero backup and a Terraform state write happened simultaneously.

Velero reads Garage's S3 API to enumerate backup objects. Terraform reads the same database to verify state file existence. Both hit SQLite through Garage's metadata layer. Under NFS, the concurrent reads triggered the lock delegation failure, and SQLite corrupted pages 169–184 of db.sqlite.

The Recovery

SQLite has a .recover command that can extract data from a corrupted database:

# Dump recoverable data from corrupted SQLite
sqlite3 db.sqlite ".recover" > recovered.sql

# Recreate the database from the dump
sqlite3 db_clean.sqlite < recovered.sql

The .recover command scanned every page of the corrupted file and extracted whatever data it could read. For pages 169–184 (the corrupted range), it found partial data — enough to reconstruct the bucket and key metadata, but not enough to guarantee referential integrity.

After recovery, the missing objects (terraform-state bucket and Atlantis lock key) had to be re-inserted manually via Python:

import sqlite3
import msgpack

conn = sqlite3.connect("db_clean.sqlite")
# Garage uses msgpack-encoded metadata with 'G2key'/'G2bkt' prefixes
# Re-insert the terraform-state bucket
conn.execute("INSERT INTO buckets (name, ...) VALUES (...)")
conn.commit()

The data was restored, but the trust was gone. The database could corrupt again under the same conditions.

The Fix

Move every embedded database to local-path:

# Garage — meta volume moved to local-path
volumes:
  - name: meta
    persistentVolumeClaim:
      claimName: garage-meta      # NOW: local-path (was: nfs-client)

The apps that need local-path:

App	Database	Why local-path
Garage S3	SQLite (metadata)	File-locking requirements
Mealie	SQLite (recipes)	WAL mode + concurrent access
Home Assistant	SQLite (state)	Inotify-based DB writes
Authelia	PostgreSQL (CNPG)	CNPG manages its own PV

PostgreSQL (via CNPG) doesn't have the SQLite lock problem because it uses its own file locking, but CNPG requires local-path or a CSI driver that supports ReadWriteOnce — NFS's ReadWriteMany semantics can confuse CNPG's WAL archiving.

The Trade-off

local-path means the PVC is pinned to one node. If the pod reschedules, it loses access to the data. For Garage, this is acceptable: Garage runs on a single node, and if that node goes down, the S3 data is unavailable regardless (it's on the same host).

For databases that need HA (Postgres, Redis), the solution isn't local-path or NFS — it's a managed operator (CNPG for Postgres) that handles replication and failover independently of the storage layer.

The principle: if the application uses file-level locking (SQLite, BoltDB, LMDB), it goes on local-path. If it uses network-level locking (PostgreSQL, MySQL), it goes on NFS or a managed operator.

SQLite on NFS is the same failure mode as running SQLite on an SMB share in a Windows domain: the file-locking semantics are fundamentally incompatible. In Azure, this maps to Azure Files (SMB-backed) vs. Azure Disk (block storage). Azure Files supports SMB locks but has the same delegation latency issues under concurrent access — any application that needs tight file-level locking should use Azure Disks, not Azure Files. The principle is identical: embedded databases need local, low-latency storage with native file-locking support.

Designing Data-Intensive Applications* covers exactly this class of correctness assumption — what a storage layer actually guarantees about concurrent access versus what an application silently assumes it guarantees — in far more depth than a corrupted db.sqlite file teaches you in the moment.