ZFS dataset shadowing hid 5.5TB from me during an outage

bishwajit mandal — Wed, 16 Sep 2026 05:39:05 +0000

My reverse proxy container stopped starting one evening, which meant every service behind it stopped resolving. Jellyfin, Immich, Nextcloud, the dashboard — all of it sits behind Nginx Proxy Manager in CT104, so when CT104 stays down, the whole *.bmlab.co.in namespace is down with it.

It took me a few hours to find the actual cause, and the thing that cost me the most time was a warning that had nothing to do with it. Writing it up because the final root cause is a ZFS behaviour I hadn't run into before, and because I made a couple of wrong calls along the way that are worth showing.

The setup

A single mini PC running Proxmox 8.x. Seven LXC containers and one VM:

CT100 — NAS-ish container, Samba and Docker
CT101 — *arr stack
CT102 — Immich
CT103 — Nextcloud
CT104 — Nginx Proxy Manager, Homarr, cloudflared
CT105 — Jellyfin
CT106 — Pi-hole
VM108 — Tailscale exit node

Two ZFS pools: flash, a 2TB SSD, and vault, a single 8TB IronWolf holding media and downloads. Split-horizon DNS via Pi-hole, wildcard Let's Encrypt cert terminated at NPM.

One host. No redundancy. That matters later.

The red herring

pct start 104 failed. The log had an AppArmor line in it — a complaint about lxc.apparmor.profile — sitting near the top of the output, which is exactly where you look first.

I spent real time on that. Profile syntax, whether unconfined was still valid on this kernel, whether a Proxmox update had tightened something. All of it was wasted. The warning was noise that had been in that config for a long time and had never stopped the container before.

The lesson I keep re-learning: a warning that is present during a failure is not the same as the cause of the failure. The question to ask is "was this here yesterday, when it worked?" It was.

The actual cause

Further down, past the AppArmor line, the mount of the container's rootfs was failing.

CT104's rootfs was a raw image file on vault, mounted through a loop device with ext4 on it. ext4 has a feature called multi-mount protection (MMP), which exists to stop two hosts mounting the same shared filesystem at once. It works by writing a heartbeat block to the filesystem on mount and re-writing it periodically.

The mount was failing at that write.

Not because of any shared-storage conflict. Because there was nowhere to put it:

# zfs list vault
NAME    USED  AVAIL  REFER  MOUNTPOINT
vault  7.27T    36M   ...   /vault

36 megabytes free. A filesystem that cannot write a single block cannot complete an ext4 mount, so the container could not start. The whole domain went down because a pool hit 100%.

Why the pool was full

Two causes, both mine.

Sparse raw images over-provisioned past the pool. CT100 and CT104 both had raw disk images on vault. Their apparent sizes added up to roughly 9.24T on a pool with 7.27T usable. Sparse allocation means this is allowed right up until the moment it isn't, and nothing warns you on the way there.

Downloads were being copied, not hardlinked. Sonarr and Radarr were configured such that an imported file became a second full copy rather than a hardlink to the original. About 2.4T of the pool was the same media twice.

The part I actually want to write about

Here is where it got strange. I went looking for what was consuming the space, and du on the obvious paths didn't account for anywhere near the full pool.

/vault/media/movies looked nearly empty. So did /vault/media/tv and /vault/downloads/complete. But the pool insisted the space was gone.

The cause was dataset shadowing. I had created child datasets:

vault/downloads
vault/downloads/complete
vault/media
vault/media/movies
vault/media/tv

Each child mounts at its own path inside the parent's directory tree. If content was written into /vault/media/movies before the child dataset vault/media/movies was created and mounted over it, that content still exists — it lives in the parent dataset, underneath the mountpoint. Once the child is mounted on top, the parent's copy is invisible. Every normal tool, du and ls and ncdu included, sees the child dataset and reports what's in the child.

About 5.5TB of real content was sitting underneath those mountpoints, occupying space, unreachable by path.

To see it, you bind-mount the parent dataset somewhere else. The bind exposes the parent's own directory tree without the children mounted over it:

mkdir -p /mnt/raw-dl
mount --bind /vault/downloads /mnt/raw-dl
du -sh /mnt/raw-dl/*

Important detail I got wrong first: you bind the parent dataset, not the child. Binding vault/downloads/complete just shows you the child again. Binding vault/downloads shows you what the parent holds at that path with nothing shadowing it.

I will also admit that mid-debug I talked myself out of this diagnosis and decided the shadowing wasn't real, because the child datasets looked fine when I checked them directly. That was the wrong call and it cost me another pass. The shadowing was real. It's just that you cannot see it from the path you'd naturally check.

Two more things I got wrong

du numbers on sparse files are apparent size, not allocated blocks. I estimated how much I'd reclaim by deleting things, and the estimate was badly off against what ZFS actually freed. For sparse images, du --apparent-size and reality diverge a lot. Check zfs list after deleting rather than trusting arithmetic done in advance.

I had never taken a backup. More on that below.

Recovery

In order:

1. Free space. Deleted verified duplicate downloads — the ones that had a matching file already in the library. Several hundred gigabytes of TV, plus some remuxes I had twice. Also found an .exe sitting in a downloads folder that had no business being there. Got vault to 726G free, 90% used.

2. Move rootfs volumes off the pool that filled. CT100 and CT104's root disks went from directory storage on vault to ZFS subvolumes on flash:

pct move-volume 104 rootfs flash
pct move-volume 100 rootfs flash

The reverse proxy no longer depends on the media pool having free space. That was the single most valuable change of the night.

3. Stop backing up the media mounts. CT100 had a 7850G mount point and a 200G one, both flagged for backup, which makes any vzdump attempt absurd. backup=0 on both in the container config.

4. Create a backup target and actually use it.

zfs create -o refquota=400G flash/backups

Added as directory storage flash-backups in Proxmox, then ran vzdump for CT100, CT103, CT104 and CT106.

Those were the first backups this system has ever had. It had been running for over a year. I had built split-horizon DNS, a wildcard cert, a Tailscale exit node and a seven-container stack, and had never once run vzdump. That is the real finding of this whole incident.

5. Removed the AppArmor unconfined line from CT104's config, since it was doing nothing except drawing my attention during an outage.

Verifying it end to end

Rather than trusting the browser and its cache, I checked the proxy path directly:

curl -I --resolve jellyfin.bmlab.co.in:443:192.168.0.205 \
  https://jellyfin.bmlab.co.in

HTTP/2 302 back from openresty, wildcard cert valid. That confirms DNS resolution, TLS termination and the proxy host entry in one call, without any client-side DNS in the way.

Still open

Deliberately listing these unfinished, because the writeup isn't the fix:

Hardlinks are still not enabled in Sonarr/Radarr. Settings → Media Management → Use Hardlinks instead of Copy. Until that's on, the 2.4T problem regenerates.
No qBittorrent auto-removal by ratio or seed time yet.
vzdump is not on a schedule. I ran it by hand.
Some files in vault are owned by a UID outside the unprivileged container's mapping range and need fixing.
The now-empty child datasets need destroying once I've confirmed nothing lives in them.

What I'd tell past me

Alert on pool capacity. A threshold at 80% would have turned this outage into a Tuesday afternoon task.
Don't put an infrastructure container's rootfs on your bulk storage pool. The service that fixes everything else should not depend on the pool most likely to fill.
If du and zpool disagree by terabytes, suspect shadowed datasets before you suspect either tool.
Create datasets before you write data into the path, not after.
Take a backup. I know. I know.

DEV Community: bishwajit mandal

ZFS dataset shadowing hid 5.5TB from me during an outage