The same outage that suspended the first pool also took down a second pool on the same box in a different, harder way. That's the piece the suspended-pool recovery closed by pointing at. This is that piece.
The first pool showed multiple disks marked as faulted, but the vdev structure was intact. It was a path issue that looked catastrophic. The second pool, on the same box, hit by the same outage, showed something worse: an entire RAIDZ1 vdev in the UNAVAIL state, with every disk in that vdev marked as REMOVED. The other vdev in the pool was fully ONLINE. But that didn't matter.
A ZFS pool striped across multiple vdevs is only as available as its least-available vdev. If one vdev is unavailable, the pool is unavailable, even if every other vdev is healthy. Pool-level parity does not protect against a whole vdev going away — that's not what ZFS's striping model does. If you want the pool to survive one vdev disappearing, you need mirror-of-vdevs or dRAID or another topology that plans for that specific failure. RAIDZ1 across vdev-striping does not.
I refer to the affected pool as MediaPool-B, matching the naming from the sibling piece.
The recovery for this pool was fundamentally different from the first one's. The first pool's recovery was zpool clear after checking the disks were visible. This pool's recovery required reading the ZFS labels directly to confirm the "removed" disks still had valid pool metadata — before running any command that could touch state. And after the pool came back online, there was a second problem waiting: several hundred files with permanent errors that had to be handled cleanly before the pool was truly clean again.
// what the console showed
The pool's status output, structurally:
MediaPool-B SUSPENDED
raidz1-0 UNAVAIL
disk 1 REMOVED
disk 2 REMOVED
disk 3 REMOVED
disk 4 REMOVED
raidz1-1 ONLINE
disk 1 ONLINE
disk 2 ONLINE
disk 3 ONLINE
disk 4 ONLINE
disk 5 ONLINE
disk 6 ONLINE
Two RAIDZ1 vdevs in the pool. One fully online. One showing every disk as REMOVED and the vdev itself as UNAVAIL. The pool as a whole in SUSPENDED state.
The visual read of that output is that half the pool has evaporated. The instinct is to look at the TrueNAS GUI, see disks appearing as "unassigned," and reach for either "Add to Pool" or "Wipe." Both are wrong. Both are, in fact, the most dangerous things you can do in this situation.
The reason is a distinction the TrueNAS SCALE GUI doesn't surface clearly. A disk shown as "unassigned" in the GUI is a disk that TrueNAS is not currently associating with a pool through its own tracking. That's not the same as a disk without a ZFS label on it. Every disk that was ever a valid ZFS pool member has labels on it — four of them, redundantly placed at the start and end of the device — describing the pool it belongs to, the vdev it was part of, and the identifier that makes it a member. Those labels survive the pool being suspended. They survive TrueNAS forgetting about the pool. They survive a reboot. What they don't survive is you wiping the disk, or adding the disk to a new pool structure. Both destroy the labels. Neither is recoverable.
So the first correct move is exactly the one the GUI is designed to push you away from: don't touch the disks yet.
// the critical move: don't trust the GUI
The right move is to read the ZFS labels directly, before any GUI action, to prove those "unassigned" disks are still legitimate members of the pool that ZFS thinks is broken. The tool is zdb, ZFS's low-level debugger, and the flag is -l for label inspection.
The pattern I used was a shell loop that scanned every partition on every disk, called zdb -l on each, checked whether the label mentioned the pool by name, and printed the relevant identifying fields for any match:
for d in /dev/sd*2; do
zdb -l "$d" 2>/dev/null | grep -q "MediaPool-B" && \
echo "===== $d =====" && \
zdb -l "$d" 2>/dev/null | egrep "name:|guid:|top_guid|path:"
done
The output came back with entries for the four disks that had been shown as REMOVED. Each entry had a name: field matching MediaPool-B, a guid: uniquely identifying that member disk, and a top_guid: field identifying which vdev the disk belonged to. The four top_guid: values matched each other — meaning the four disks did all belong to the same vdev — and matched the top-vdev GUID of the missing raidz1-0.
That was the turning point of the recovery. The four disks were not gone. They were not damaged. They were not the wrong disks. They were exactly the four members of the missing vdev that they'd always been, still present with their labels intact, waiting for ZFS to resume the pool.
This is the class of check that matters at the moment when the GUI's information is missing the shape of the actual state. zdb -l doesn't care what TrueNAS thinks the disks are for. It reads what's actually written on the disk. That distinction — between what a management layer says about a disk and what the disk itself says about its history — is the whole point of ZFS's label design. Nothing at the storage layer should trust the management layer's model of state over the storage layer's own record.
If any of the four disks had failed to show a matching label, or had shown a label for a different pool, the response would have been different. In that case, the correct move would be to stop, escalate to the physical layer, and only come back to software recovery after understanding why a disk was reporting inconsistently. Nothing was inconsistent. Every disk reported exactly the label expected. The pool was recoverable.
// zpool clear, this time with confirmation
With the four missing disks confirmed as legitimate vdev members via their labels, the recovery command was the same one the first pool's recovery used:
zpool clear MediaPool-B
zpool status -v MediaPool-B
The pool came back to ONLINE. All disks online. READ, WRITE, and CKSUM counters at zero across every device. The vdev that had been UNAVAIL was now healthy. The pool was importable, mountable, and — critically — the labels told ZFS the same story the labels had always told, so the pool's history and identity carried through cleanly.
This is where the confidence delta from the first pool's recovery mattered. In the first pool, zpool clear was applied after confirming the disks were visible and the kernel path was quiet. Here, it was applied after confirming the disks weren't just visible — they were still legitimate members of the pool ZFS was trying to bring back. Different level of certainty, different level of stakes. A blind zpool clear on the first pool's shape would have worked. A blind zpool clear on this pool's shape, without the label check first, would have been a bet on state that hadn't been verified.
But zpool status after the clear showed something the first pool didn't produce.
// the second problem: permanent errors
The pool was ONLINE. Every disk was ONLINE. Every counter was zero. And ZFS was reporting permanent errors in application data — specifically, several hundred files that had errors ZFS couldn't reconcile from parity.
Permanent errors mean exactly what they sound like. ZFS has tried to reconstruct these files from the parity blocks in the vdev, and the reconstruction can't produce the file the pool's metadata says should exist. The file's content is unrecoverable at the ZFS layer. What you do next depends entirely on what the files are.
For this recovery, the affected files were regeneratable workload data — application-layer files that were expensive to reproduce but not irreplaceable. If they'd been irreplaceable — family photos, business documents, source code without a git remote — the correct next step would have been to restore from backup for those specific paths, then clear the error state. Application context determines the correct operational response. There is no single answer.
The sequence for regeneratable data:
Save the evidence first. The zpool status -v output lists every affected file by path, and that list is the ground truth of what's actually corrupt. It should be captured to disk before anything is deleted, so the record survives whatever comes next:
zpool status -v MediaPool-B > /root/MediaPool-B_corrupt_files_$(date +%F_%H%M).txt
Extract just the file paths, filtered to the affected dataset:
zpool status -v MediaPool-B \
| grep "/mnt/MediaPool-B/Dataset/" \
> /root/MediaPool-B_corrupt_delete_list.txt
Count and preview before doing anything destructive:
wc -l /root/MediaPool-B_corrupt_delete_list.txt
head -20 /root/MediaPool-B_corrupt_delete_list.txt
In this case, the delete list came out to 311 files, every one of them under the expected dataset path. That confirmed the impact was fully contained to the regeneratable workload before removing anything.
Delete the corrupted files with whitespace trimming — the zpool status output leaves leading spaces on file paths, and rm doesn't handle those well:
while read f; do
f="$(echo "$f" | xargs)"
rm -v "$f"
done < /root/MediaPool-B_corrupt_delete_list.txt
Then verify none of the listed paths still exist:
while read f; do
f="$(echo "$f" | xargs)"
[ -e "$f" ] && echo "$f"
done < /root/MediaPool-B_corrupt_delete_list.txt | wc -l
Zero. All 311 files gone.
Clear the error state and scrub:
zpool clear MediaPool-B
zpool scrub MediaPool-B
The scrub is what actually retires the stale error header. zpool status immediately after zpool clear will still show the permanent-errors block with no files listed — the header persists until a scrub confirms all data is clean. After scrub completed, the pool status came back to:
MediaPool-B = ONLINE
scrub completed with 0 errors
errors = No known data errors
// what SMART said afterward
Before declaring the recovery complete, I checked SMART health on each of the four disks that had been marked REMOVED. Any of them could have had a real physical issue that contributed to the outage's severity on this pool.
smartctl -a /dev/diskA
smartctl -a /dev/diskB
smartctl -a /dev/diskC
smartctl -a /dev/diskD
Three of the four came back clean — SMART health OK, normal temperatures, zero uncorrected errors, zero reallocated sectors. Fourth disk came back with a small grown-defect count. Not zero, but low enough that immediate replacement wasn't the answer. That disk went on a watch list to be checked at each subsequent scrub, with the plan to pull it if the count grew.
The lesson here is a small one but worth stating: recovery doesn't end at "the pool is back." It ends at "the pool is back and the drives that were involved in the incident have been individually checked for physical health." Neither one substitutes for the other.
// what I'd make instinctive
Four things went into the runbook after this pool.
Unassigned in the GUI is not the same as unused. TrueNAS SCALE shows disks as "unassigned" when its own tracking doesn't currently associate them with a pool. That has almost nothing to do with what's on the disk. A disk that's shown as unassigned can still hold valid ZFS labels for a pool the system is trying to bring back. The GUI's model of state is separate from the storage layer's model of state. When those diverge, trust the storage layer.
Read ZFS labels before doing anything destructive. zdb -l is cheap to run, always available, and reads exactly what's written to the disk. It doesn't rely on any pool being imported, any middleware being healthy, or any GUI showing correct information. When something looks catastrophically wrong with a vdev, zdb -l is the first tool to reach for, not the last. Wiping a disk that still has valid pool labels is destructive and not recoverable — thirty seconds of label inspection prevents that class of mistake entirely.
One unavailable vdev takes the whole pool down. If the pool is striped across multiple vdevs — the default for capacity-oriented ZFS deployments — the pool's availability is the availability of its worst vdev. Mirror-of-vdev or dRAID topologies are the alternatives when one whole vdev going away isn't an acceptable failure mode. Choose the topology deliberately, and know what failure modes you've bought into.
zpool clear doesn't guarantee no permanent errors. The pool coming back online means ZFS has decided the vdev structure is workable. It doesn't mean every file survived. Permanent-error handling is a separate operational step, and the correct response depends entirely on whether the affected files can be regenerated or need to be restored from backup. There is no single answer that's right for every dataset.
None of these are theoretical. Every one of them was learned by doing the work right the first time — or nearly wrong once, which counts for the same lesson.
// closing
Two pools, two recoveries, one outage. The first pool was a path issue that looked like a shelf failure — zpool clear after confirming the disks were visible. The second pool was a vdev issue that looked like half the pool had evaporated — zpool clear after confirming the labels were still valid, then several hundred files deleted, then a scrub. Different failure modes, different recovery sequences, different mental models. Both pools came back without any physical replacement.
The unified lesson across both pieces is small but load-bearing: ZFS's failure reporting is designed to be interpreted, not complied with. The console output tells you what ZFS observed. It doesn't tell you what actually happened, and it doesn't tell you what to do about it. That interpretation is the operator's job, and it depends on understanding what the storage layer is actually asking of you at that moment.
The ongoing storage roadmap for this rack starts with backup posture — the PBS project referenced in the CORE-and-SCALE piece is what turns "recovered without backup" from operator luck into operator design. Recoveries this clean shouldn't have depended on the outage being transient. That's a gap I'm closing, not a story I'm telling as a success.
If you're running mixed-workload TrueNAS today and haven't tested a backup restore recently, that's your homework. It's mine too.
Top comments (0)