The plan was a network change with a storage flavour. Move a Windows workload and two TrueNAS servers off the legacy flat network and onto the new dual-TOR fabric on the workload VLAN. Three cutovers. Repoint the SMB mappings. Done.
The workload moved. Both TrueNAS servers moved. The mappings got repointed. Everything the change plan named came back healthy.
What the plan didn't cover was what "healthy" meant at each layer of the storage stack — because there isn't one layer. There are at least five, they can each be independently green or broken, and a healthy signal at one of them does not imply a healthy signal at the next. That's the piece.
Worth saying up front: I did not know the migration was going to teach me this while it was happening. During the incident, each failure looked like its own problem. It was only after everything was back that the pattern surfaced. Five different failure modes, five different layers of the storage stack, five different validation steps that would have caught each one independently. The layered framing is retrospective. If it reads as if I was tracking it that way in real time, that's an artefact of writing about it, not what the day felt like.
The five layers, in the order they surfaced during the migration:
- Network reachability — physical link plus L2/L3 configuration
- Service reachability — the SMB advertisement, independent of what's behind it
- Filesystem state as the middleware sees it — TrueNAS's cached view of pool status
- ZFS state as the kernel sees it — pool importable, disks readable, labels intact
- Physical storage-path state — the SAS and CAM layer under ZFS
Each of the five failed at least once during the day. Each of them failed in a way that would have been invisible from the layer above.
// layer 1: network reachability
The Windows workload cutover was the first to surface a network-layer failure. Old address on the legacy fabric, new address on the workload VLAN, TOR-B. Cable in, port configured, physical layer up. The switchport showed a valid 10G link. Windows showed the interface as Disconnected.
That's the layer-1 problem in one sentence. The physical link doesn't prove the operating system has a usable NIC.
The workload host had a 10G adapter installed but no driver loaded for it. The only interface Windows recognised was the onboard Intel I219-LM 1G. From the switch's side everything looked correct. From the workload's side there was no interface to bind an address to. Both views were internally consistent. Both views were incomplete.
The fix was the standard one — install the correct 10G adapter driver, watch the interface enumerate in Windows, watch the switchport carry the traffic that had been waiting for it. The lesson isn't the fix. The lesson is that the switchport LED and the OS network stack are two different signals, and only the OS signal proves the host can actually use the link. On the migration checklist "link up" needs to mean link up on the host's side, not the switch's side.
The second network-layer failure surfaced when I tried to reach one of the TrueNAS servers on its new address. It didn't respond. Ping timed out. RDP timed out. The workload couldn't enumerate its shares because it couldn't reach the server at all.
I don't remember the specific moment that turned into an investigation — I might have gone hunting on the switch first, I might have gone straight to the server. What I found was that the workload VLAN hadn't been added to the LAGG on the TrueNAS server. The switch was configured. The physical link was up. The other end of the LAGG on the server had no idea the workload VLAN existed as a tagged interface.
Adding the workload VLAN as a tagged interface on the existing LAGG, with the server's new address on it, brought the server online. From the workload side, the SMB shares immediately became reachable.
Both of these are "network" failures in the loose sense, and both had the same shape: healthy signal on one side of the link, invisible break on the other. Layer 1 validation isn't "the switch shows the link up." Layer 1 validation is "both endpoints can talk to each other at the address the change plan says they should." Every migration touching L2 or L3 configuration needs to check both directions before declaring the layer green.
// layer 2: service reachability
With the network layer working, the workload tried to enumerate its SMB shares. Most of them appeared. One of them — the largest one, the primary storage target for the workload — returned a Windows error: System error 67 has occurred. The network name cannot be found.
That message is the operating system telling you the SMB server accepted your connection but refused to serve you the share you asked for. It's not a network failure. It's not an authentication failure. It's the SMB service saying yes, I know that share name, and no, I can't give it to you right now.
The dataset backing that share didn't exist.
Specifically: the share was advertised by TrueNAS's SMB configuration, correctly. But the ZFS dataset the share was configured to expose wasn't mounted. It wasn't mounted because the pool holding it wasn't imported. It wasn't imported because the pool's disks weren't visible to the operating system. And the disks weren't visible because the JBOD shelf holding them hadn't been powered on after the re-rack.
The layer here is service reachability — SMB advertising a share is one thing, the share having a valid backing dataset is another thing entirely. TrueNAS doesn't disable an SMB share when its backing dataset goes away. The share stays advertised. The share stays in the configuration. Windows can see it in the list of shares on the server. It just can't be used, and the error message it produces is deliberately unhelpful about why.
Powering the JBOD brought the disks back to visibility. But that only fixed the physical prerequisite. The middleware layer above ZFS still had its own idea of what was going on, and that's the next layer.
// layer 3: middleware view vs ZFS reality
With the JBOD powered, camcontrol devlist on the TrueNAS server showed all the drives. zpool import — the command that lists importable pools — showed the three pools from that shelf as ONLINE and ready to import. From the ZFS layer's perspective, the pools were healthy and available.
The TrueNAS GUI disagreed. Every one of those three pools was still shown as OFFLINE, in the state the middleware had recorded when the JBOD went dark. From the middleware's view, nothing had changed since the shelf went off.
That's the third layer. TrueNAS caches pool state in its middleware database. When something happens outside the middleware's visibility — a JBOD losing power, a SAS cable being pulled, a controller resetting — the middleware doesn't automatically catch up when the situation resolves. The kernel's view of ZFS updates in real time. The middleware's view updates when it's told to.
The recovery for each pool was the same sequence. Export/Disconnect from the GUI, being careful not to check the destroy-data or delete-configuration boxes. Then re-import via the GUI's Import Pool workflow. The pool would enumerate correctly, the middleware would record it as ONLINE with its current disks, and the shares configured against its datasets would start serving. Three pools, three cycles of Export/Import, three shares recovered.
The middleware-vs-ZFS divergence isn't a bug in TrueNAS. It's a consequence of the middleware being a caching layer over the kernel-level truth. The important operator move is knowing that the two views can drift, and knowing which one to trust when they disagree — the answer, always, is the kernel. If zpool import says the pool is available and the GUI says it's offline, the pool is available.
// layer 4: ZFS state under SAS path interruption
The second TrueNAS server surfaced the piece's first real ZFS-layer failure. Several SMB mappings from the workload connected fine but couldn't enumerate files. Timeouts, empty directory listings, hangs on read.
I opened the GUI and looked. One of the pools — call it Pool-A — was in the UNAVAIL state, with all eight of its members marked REMOVED.
That's a scary status line. It suggests the pool has lost every one of its physical members simultaneously. If that were true it would mean the array had suffered a catastrophic multi-drive failure, and the response would be one of the worst calls an operator makes: try to import from backups, accept whatever data loss the last replication window implies, rebuild.
It wasn't that. It looked like that. Because the pool had lost every disk at once, the diagnostic instinct that reaches for "eight independent drives don't fail simultaneously" needed to fire before the instinct that reaches for "restore from backup." A whole pool showing all members REMOVED at once is almost always a path problem, not a media problem. And a path problem is recoverable at the software layer if the labels on the disks are still intact.
I checked. camcontrol devlist showed all eight drives. Their GPTIDs — the ZFS-tracked partition identifiers — had reappeared. zdb -l on each of the drives showed valid ZFS labels naming Pool-A as their pool. SMART data on each of the eight came back clean. Nothing about the disks themselves said they were unhealthy.
The pool's UNAVAIL state was a stale view. zpool clear on Pool-A brought it back ONLINE, all eight members ONLINE, no permanent errors. From the ZFS layer's perspective, the pool was fully recovered.
Except SMB access from the workload was still broken.
The layer-5 lesson to come is that below ZFS is the SAS and CAM path, and that's the layer that had interrupted. The layer-4 lesson is smaller but easy to miss: pool state and consumer state can diverge. Recovering the pool at the ZFS layer doesn't automatically recover the services consuming that pool. Samba, in particular, holds ZFS VFS handles that go stale when the pool goes away and don't refresh when the pool comes back. From Samba's point of view, the dataset it was serving still didn't exist, even though zpool status said it did.
Restarting the SMB service — service samba_server restart from the shell, or Stop/Start on the SMB service in the GUI — dropped the stale handles and rebuilt them against the newly-recovered pool. Windows access came back immediately. Layer 4 clean.
// layer 5: physical storage-path state
The last failure, and the hardest, was on the same TrueNAS server. A second pool — call it Pool-B — was also showing UNAVAIL, also with all eight members REMOVED. The diagnostic sequence I'd just run on Pool-A said the same story: eight members visible in camcontrol devlist, ZFS labels intact, SMART clean. Same recovery expected.
zpool clear on Pool-B did nothing. The pool stayed UNAVAIL. zpool reopen — the command that tells ZFS to re-scan the vdev's devices — did nothing. zpool online on individual members did nothing. None of the ZFS commands could bring the pool back.
At this point I looked at the kernel logs. dmesg was full of messages I hadn't seen on Pool-A. CAM Periph destroyed events. SCSI transport errors. The CAM layer — FreeBSD's Common Access Method, the driver layer that sits between ZFS and the SAS host bus adapter — was in a state where it had torn down its representation of Pool-B's devices and couldn't rebuild them without a full reset.
Then the situation started to spread. Other pools on the same TrueNAS server started showing UNAVAIL in the GUI, one after another. zpool status on any of them returned slowly, then hung. Trying to interact with the affected pools started hanging the shell that ran the commands. The system was becoming unresponsive.
This is a below-ZFS problem. ZFS can only manipulate what CAM can present to it, and CAM can only present what it can reach through the SAS layer. When the SAS/CAM path wedges, ZFS commands become impossible not because there's anything wrong with ZFS but because the layer below it has lost track of what devices exist. There is no ZFS-level command that fixes a stuck CAM layer.
The clean recovery, once I'd worked out what I was looking at, was blunt: stop SMB (to prevent consumer-side hangs during the reset), hard power off TrueNAS1, fully power-cycle both attached JBOD shelves, let everything settle for a couple of minutes, then power the server back on. That resets the entire storage path from the server's HBA outward — the SAS controller re-enumerates the shelves, the shelves re-enumerate their drives, CAM builds a fresh representation from scratch, and ZFS gets a stable substrate to work against again.
After the reset, both TrueNAS servers came back with all their pools ONLINE. Neither the drive labels nor the ZFS state had actually been damaged by any of this. What had failed was the layer under both of them.
The layer-5 lesson is the one that took me longest to internalise: below ZFS is a stack of physical-path components — SAS controller, cables, JBOD backplane, drive interfaces — and when any of those wedge, no amount of ZFS-level work can fix the situation from above. zpool clear cannot repair a hung CAM state. zpool reopen cannot re-enumerate devices the driver layer has torn down. Sometimes the only recovery is a full storage-path reset, from the HBA outward, and that means the server comes down for the duration.
The corollary is a diagnostic one: when zpool commands start hanging on a pool whose disks are physically present and healthy, don't reach for more ZFS-level tools. Look at CAM. Look at the kernel logs. Look for signs that the layer below ZFS has become the problem — because when it has, ZFS is the wrong tool for the job.
// what the migration design did right
The five failures across the day are the bulk of the piece, and worth being clear-eyed about: none of them were caused by the migration design being wrong. Every one was an existing condition or a physical-layer artefact that the fabric change happened to surface.
What the design did right was containment. The workload VLAN on the new fabric isolates this workload's storage traffic from everything else running on the rack. The TOR-A/TOR-B split gives each end of the LAGG its own upstream. The workload servers and the storage servers on the new fabric can talk to each other. Nothing on the legacy network was affected by any of the failures above.
That containment is the reason a bad day for the workload's storage stack was not also a bad day for anything else on the rack. The Kubernetes cluster kept running. The other DePIN workloads kept running. The rack's other services didn't notice the migration was happening beyond the fact that traffic patterns shifted subtly as the workload's dependencies moved. That's not a lucky outcome. It's what per-service VLAN segmentation is for, and it worked as designed.
// what I'd make instinctive
Five doctrine points came out of the day, one per layer.
Layer 1 — network reachability. Switch-side signals and host-side signals are two different things. The switchport LED tells you the link is electrically up; only the host OS's network stack tells you the link is usable. Every migration checklist item that says "verify link" needs to be explicit about which side is being verified. Both sides. Always.
Layer 2 — service reachability. SMB will happily advertise a share whose backing dataset is missing, and the error message clients get is deliberately opaque. Validating an SMB migration means validating that each share can not only be seen but can also be opened and read from — because "visible in the share list" and "backed by a real dataset" are two different states.
Layer 3 — middleware view vs kernel truth. When a management layer caches state over a kernel-level source of truth, the cache can drift. Export/Import in the TrueNAS GUI is the standard way to refresh a pool's middleware view after something upstream has changed. Recognise the situation early — GUI shows OFFLINE, zpool import shows ONLINE — and don't waste time trying to fix the middleware view through the middleware. Cycle it.
Layer 4 — ZFS state under SAS path interruption. A pool showing every member REMOVED simultaneously is almost always a path event, not a media event. Before touching backup recovery or destructive commands, verify the disks with camcontrol devlist, verify the labels with zdb -l, verify SMART. If the disks are healthy and the labels are intact, zpool clear is likely all that's needed. Also — don't forget that pool recovery isn't service recovery. If Samba was serving the pool, it needs to be restarted to drop stale VFS handles.
Layer 5 — physical storage-path state. ZFS can only work on what CAM can present. When CAM wedges, no ZFS command will get through. Kernel logs — specifically CAM Periph destroyed and SCSI transport errors — are the signal that the problem has moved below ZFS. Recovery may require a full physical power-cycle of the storage path from the HBA outward. Plan for that possibility on any migration involving SAS-attached JBODs.
// closing
The rack power migration piece closed on the observation that a change plan can be right and still be incomplete — the plan describes what you know, and the rack describes what actually is, and the gap between them is where changes reveal themselves. This piece is that observation transposed one axis. A layer can be green and the whole still doesn't work, because storage isn't one thing. It's a stack, and each layer of the stack has its own way of failing, its own way of appearing healthy, and its own validation step that has to be run before the layer above it can be trusted.
The migration itself finished the day it was supposed to. Both TrueNAS servers on the new fabric, all pools ONLINE, all shares serving, the workload back in operation on the new fabric. What actually got done that day wasn't just a network cutover — it was a five-layer diagnostic, retrospectively organised into a pattern I could name.
If you're running a storage migration and your validation checklist has one entry per host, you have four fewer entries than you need.
Top comments (0)