Originally published at woitzik.dev
Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.
The circularity: Velero backs up into Garage S3, which runs inside the Kubernetes cluster it's backing up. Cluster lost → Garage PVC lost → Velero backup bucket gone → nothing to restore from.
This is the backup equivalent of storing your offsite backup key in the safe you're backing up. The offsite target exists in config but isn't actually reachable — and you don't realize the gap until you need it.
View the complete homelab infrastructure source on GitHub 🐙
The Two-Problem Stack
Two issues compound into structural fragility:
Problem 1: Backup Circularity
Velero's backup storage location points to Garage, an S3-compatible object store running as a StatefulSet inside the cluster. Every Velero backup — daily snapshots of Vault, Authelia, databases, ArgoCD configs — lands in a bucket that depends on the cluster surviving.
Cluster lost → Garage PVC lost → Velero backup bucket gone → nothing to restore from
PBS backs up the VM/LXC disks to a Seagate 2TB External HDD*, but PBS runs on the same physical host — a BMAX Mini PC*. Simultaneous loss of host + USB HDD means total unrecoverability at the VM layer. Velero was the last safety net — except it was circular.
Problem 2: Torn-State Hazard
Two PVCs with different storage backends, different failure modes:
| PVC | Storage | Location | Failure mode |
|---|---|---|---|
garage-meta |
local-path |
k3s node | Node-pinned, survives if node survives |
garage-data |
NFS (nfs-client) |
NFS LXC | Survives if NFS LXC survives |
On 2026-07-12, an unclean Garage shutdown (OOM kill) left db.sqlite corrupted — the same NFS/SQLite locking incompatibility that eventually forced every embedded database off NFS. The metadata — SQLite, bucket/key ownership, access grants — and the actual object data were on different volumes with different rollback fates. A partial restore or node loss would leave one without the other, with no clean way to reconcile them.
The Four Options
Option A: Consolidate meta + data on one volume
Move garage-meta from local-path to NFS (same backend as garage-data). Both PVCs on the same storage class → same snapshot/restore fate.
Verdict: Fixes torn-state but not circularity. NFS for SQLite is proven fragile — the 07-12 corruption happened on NFS-backed data. Moving data to local-path requires hardware you don't have yet.
Option B: External off-cluster target (Cloudflare R2)
Activate the existing r2-offsite backup storage location. Velero dual-targets: in-cluster Garage for speed + off-cluster Cloudflare R2 for bootstrap.
Cluster lost → Garage gone → Velero still has offsite copy on R2
Bootstrap: rebuild cluster → ArgoCD → Velero → restore from R2
Cost: ~$0.015/GB/month on R2 (egress free). At current backup size (~5-10Gi), negligible. Risk: Near-zero — additive only, no existing backup path changes.
Verdict: Directly solves circularity. The existing R2 config is ready — needs credentials and a "go".
Option C: PBS as the primary recovery layer
Lean into PBS as the from-scratch recovery path. PBS already backs up all VMs/LXCs (including the k3s VMs that carry Garage's data). Velero becomes a convenience layer for single-app restore, not the bootstrap path.
Verdict: Already the real bootstrap path (DR Tier 1-4). Velero is Tier 5, only if Garage survived. PBS is the actual safety net. The circularity is a Velero-level problem, not a PBS-level one.
Option D: Combine B + C (what I chose)
Activate R2 offsite for Velero (breaks circularity) AND acknowledge PBS as the primary VM-level recovery layer. Velero becomes the single-app/namespace convenience tool with a real offsite target.
| Layer | What it covers | Recovery path |
|---|---|---|
| PBS | VM/LXC disks, full host recovery | Restore VMs from PBS → cluster up |
| Velero + R2 | App-level config, secrets, databases | ArgoCD → Velero → restore from R2 |
| Garage rebuild | If PVCs are lost entirely | Terraform re-creates buckets, ESO re-seeds keys |
Verdict: Covers both problems. R2 breaks circularity (zero risk, additive). PBS handles VM-level recovery (already there). Garage rebuild from scratch is the fallback if PVCs are lost — better than a partial restore that could leave metadata/data inconsistent.
The Recovery Model
Three independent layers, each covering a different failure scenario:
Scenario → Recovery path
─────────────────────────────────────────────────────
Single app crash → Kubernetes restart (self-heals)
Single PVC lost → Velero restore from R2 offsite
Full cluster lost (host survives) → PBS restores k3s VMs → Velero from R2
Host + USB HDD lost → Insurance (nothing saves this)
The key insight: no single layer has to be perfect. Each layer covers the failure modes the layer below misses. Garage's torn-state problem becomes irrelevant when the recovery procedure is "rebuild from scratch" rather than "restore from backup."
Implementation (Zero Downtime)
The R2 activation was purely additive — no existing backup path changed:
- Create Cloudflare R2 bucket (free tier)
- Populate
r2-backuplocation.ymlwith real R2 credentials via External Secrets Operator - Apply the
daily-offsiteVelero Schedule (already committed) - Run one test backup, verify with
velero backup describe - Confirm
PodVolumeBackupentries present for PVCs
The in-cluster Garage target stays. Velero now writes to both: Garage for fast local restores, R2 for bootstrap-from-scratch. If Garage is gone, R2 has everything.
What I'd Change
Implement R2 sooner. The config existed for weeks before activation. The circularity was a known risk — the "I'll do it later" mindset is what creates disaster recovery gaps.
Add a Velero backup age alert. Alert when the most recent successful backup is older than 48 hours. If backups silently stop, you want to know before you need one.
Test the full recovery path. PBS restore of k3s VMs → cluster boot → ArgoCD reconciliation → Velero restore from R2. An untested recovery path is a theory, not a plan.
Backup circularity is the most common structural flaw in self-hosted Kubernetes: the tool that protects you depends on the thing you're protecting. The fix is always the same — add an independent offsite layer. Cloudflare R2 costs nothing at homelab scale, and the R2 egress-free model means you're not penalized for actually using the restore. The hardest part isn't the implementation. It's admitting your current backup doesn't survive the scenario you built it for.
Further reading: For understanding the full recovery architecture, Designing Data-Intensive Applications by Martin Kleppmann* covers replication, backup strategies, and the failure modes that make torn-state hazards so dangerous. The Velero circularity is a specific case of the general problem Kleppmann describes: systems that depend on their own recovery path.
Top comments (1)
"The backup equivalent of storing your offsite backup key in the safe you're backing up" is the clearest statement of this failure I've read, and the part that stings is that the config looked correct the whole time.
The structural fix is that the backup target has to be outside the failure domain by construction, not by configuration. That's how we built it at Krova Cloud (I work there) — snapshots go to S3-compatible object storage separate from the bare-metal host the Cube runs on, content-addressed and deduplicated, encrypted with a key unique to each Cube. Losing the host doesn't take the history.
There's also an export path: a whole machine as a portable archive over a private time-limited link, which is the copy that doesn't depend on us being reachable.
Your torn-state point is the subtler one though. Two volumes with different rollback fates is a partial-restore generator.