DEV Community

david
david

Posted on • Originally published at woitzik.dev

How a Single Volume Was 65% of My Velero Backup and What I Almost Excluded Instead

#kubernetes #backup #nfs #velero

Originally published at woitzik.dev

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Velero's nightly backup was taking 207 seconds and backing up 33.4GB. That's normal for a cluster with 15+ PVCs — except when one volume accounts for 65% of the total bytes. The nfs-provisioner's root-mount volume was backing up the entire shared NFS export as a single blob, even though every app's individual PVC (a subdirectory of the same tree) was already backed up separately.

The fix was a one-line annotation. The investigation almost went wrong when the first hypothesis pointed at the wrong disk entirely.

The Symptom

Velero's backup stats showed a disproportionate volume:

PodVolumeBackup: nfs-provisioner-root
Namespace: nfs-provisioner
Size: 33.4GB
Duration: 207s
Enter fullscreen mode Exit fullscreen mode

Total backup for that run: ~51GB. This single volume was 65% of the total. Every other PVC was 1-4GB.

The Investigation

The nfs-provisioner pod mounts the entire shared NFS export at /persistentvolumes. Every app's PVC is a subdirectory of this tree. Velero's defaultVolumesToFsBackup: true setting means every volume the pod mounts gets backed up — including the root mount that contains all the subdirectories.

Each app's own pod also mounts its PVC subdirectory separately. Velero backs that up too. So the same data is backed up twice: once as part of the root mount, once as the individual PVC.

The First Hypothesis (Wrong)

My first guess was garage-data/immich-library — Immich's photo library, which lives on separate USB-attached disks (Seagate Portable 2TB External HDD*). Excluding it would have reduced backup size, but the USB disks aren't on the contended rpool. The backup bottleneck was the SSD, not USB. Excluding the Immich volume would have been the wrong fix for the right symptom.

The Right Fix

Verified the target before acting:

# Confirm the root-mount volume targets rpool (the contended disk)
findmnt /persistentvolumes
# /dev/zvol/rpool/nfs  ← confirmed rpool

# Confirm the individual PVCs are subdirectories of the same tree
ls /persistentvolumes/
# default-elasticsearch-data-0/
# default-immich-redis-data-0/
# default-minecraft-data/
# ... (each is a separate PVC)
Enter fullscreen mode Exit fullscreen mode

The root mount is the entire rpool. Every individual PVC is a subdirectory. Backing up the root mount redundantly backs up everything.

The Exclusion 

# kubernetes/system/nfs-provisioner/application.yml
podAnnotations:
  backup.velero.io/backup-volumes-excludes: nfs-subdir-external-provisioner-root
Enter fullscreen mode Exit fullscreen mode

One annotation. Velero skips the root-mount volume entirely. Every app's individual PVC still gets backed up through its own pod's mount.

Result 

Before: 51GB total, 33.4GB from root mount (65%)
After:  17.6GB total, 0GB from root mount
Enter fullscreen mode Exit fullscreen mode

Backup duration dropped from 207s to ~60s. No data was lost — every PVC is still backed up individually.

The Pattern

This is a common Velero gotcha with NFS provisioners:

  1. NFS provisioner pod mounts the entire export root
  2. App pods mount individual PVC subdirectories
  3. Velero backs up both — root mount AND individual PVCs
  4. The root mount is redundant because it contains everything the individual PVCs contain

The fix is always the same: exclude the root mount via backup.velero.io/backup-volumes-excludes.

How to Find Redundant Backups 

# List all PodVolumeBackups and their sizes
kubectl get podvolumebacks -A -o json | \
  jq -r '.items[] | "\(.spec.volume) \(.status.progress.totalBytes // 0) \(.metadata.namespace)"' | \
  sort -k2 -rn | head -10
Enter fullscreen mode Exit fullscreen mode

If one volume is disproportionately large and its data is already covered by individual PVC backups, it's redundant.

🚀 Azure Firewall - Enterprise Forced Tunneling Edition — €49

  • Cycle-error-free resource ordering - deploys first time, every time
  • KMS & Azure AD bypass routes - no broken Windows VMs or auth failures
  • Dynamic for_each subnet binding - scales to any number of Spokes
  • IP Group-based firewall policies - no hardcoded IP addresses
  • FQDN baseline rules for Windows Updates and core Microsoft services

Get the enterprise module →

Full source code · one-time payment · instant download

Velero's defaultVolumesToFsBackup: true is convenient but dangerous with NFS provisioners. The root mount contains everything, and Velero doesn't know it's redundant. Check your PodVolumeBackup stats — if one volume is 65% of your total, you're backing up the same data twice.

Top comments (0)