DEV Community: JJozzieTech

After a TrueNAS ISO reinstall — the two things that broke that no one warned me about

JJozzieTech — Thu, 06 Aug 2026 19:57:14 +0000

The TrueNAS SCALE machine that runs my SMB shares was in trouble. Windows clients had stopped seeing the shares reliably — intermittent at first, then completely. When I dropped to the shell to check, systemctl status smb returned something worse than a failed service: it returned a service that no longer existed. Same for nmb. Same for winbind. The service wrappers were gone.

Samba's binaries were still there. The actual smbd and nmbd executables under /usr/sbin were untouched. But the systemd units that normally wrap them, and the TrueNAS middleware that manages both, had lost too much of themselves to be repaired by unit regeneration or middleware restart. Something at the service-layer wiring had broken beyond in-place fix.

The recovery answer, once I stopped trying to patch it, was a fresh ISO reinstall and a pool reimport. That got the service layer back. What it didn't do — what nobody had warned me it wouldn't do — was fix Windows authentication or fix ACL permissions on the imported datasets. Two things I'd inherited from the reinstall broke SMB access in ways that had nothing to do with SMB itself.

This piece is the story of those two things. The SID mismatch that made Windows reject correct credentials. The missing default ACL entries that blocked recursive permission repair. And the fix sequence that restored SMB access to what it had been before the service layer disappeared.

Same box as the ZFS suspended pool piece, different failure mode. When the storage layer goes wrong, ZFS suspends the pool and tells you exactly that. When the service layer goes wrong, the layer just disappears and Windows quietly stops answering.

// why the fresh ISO was the answer

The service layer on TrueNAS SCALE isn't just Samba. It's the systemd units that manage Samba. The middleware plugins that TrueNAS uses to expose SMB configuration through its GUI. The internal API endpoints that the middleware calls when you edit a share in the interface. All three layers were compromised.

I tried the standard recovery moves first. Regenerating service units through the middleware. Reloading systemd. Restarting middlewared to force the internal API to reinitialise. None of it took. The problem wasn't a single missing piece — it was structural damage across the middleware, systemd, and Samba integration layers that had drifted enough that regeneration couldn't reconstruct working state from what remained.

That's the moment where the answer stops being surgery and starts being a reinstall. Not the emotional decision — the reasonable one. When a service layer's structural integrity is gone, patching pieces individually just produces a service layer that limps. A fresh ISO reinstalls all three layers cleanly, at their intended versions, in their intended configuration. Fifteen minutes of install, a pool reimport, and the service layer is back to what it should be.

That's what happened. Fresh ISO. Reimport of pools. smb.service, nmb.service, winbind.service all back. midclt smb.status returning cleanly. The GUI managing SMB again without errors.

Then the actual story of this piece started.

// the two things that break after reinstall

With the service layer working again, Windows still couldn't authenticate to the shares. Correct credentials, rejected by every share on every dataset. The problem wasn't SMB anymore.

The SID mismatch.

When TrueNAS SCALE reinstalls, every user account has to be recreated. Same usernames, same passwords, same group memberships — nothing prevents you from recreating them exactly. But Windows doesn't care about your username. What Windows cares about is the SID.

A SID is the Security Identifier that Windows uses to actually attach permissions to identities. Usernames are human-readable labels layered on top; the identifier that gets written into ACL entries is the SID. When TrueNAS creates a user, it generates a SID. When TrueNAS is reinstalled and the same user is recreated, a new SID gets generated. Not the same one. The SID is not derived from the username.

Existing ACL entries on the datasets — the ones from before the reinstall — reference the old SID. Windows looks at those ACL entries, matches its authentication attempt against the SID recorded there, sees no match against the new user's new SID, and rejects the authentication with "Credentials do not match."

The credentials do match. The SIDs don't. Windows is telling the truth in a way that doesn't help you fix it.

The missing default ACL entries.

The second problem shows up when you try to fix the first one by applying corrected permissions recursively. TrueNAS SCALE's ACL editor supports recursive apply — set permissions on a parent dataset, tell it to apply down through everything below, and it walks the tree. That's the normal repair for permission drift.

Recursive apply on imported datasets errors out immediately:

Error: dacl -- Default ACL entries are required in order to apply ACL recursively

That message is opaque unless you know what a default ACL entry is. Access ACL entries govern the files and directories that exist right now. Default ACL entries govern what permissions get inherited by new files and directories created inside a parent. On a healthy TrueNAS dataset both are set. On a dataset imported from a pool after reinstall, only the access entries survived the import — the default entries didn't come across.

Without default entries, recursive apply doesn't know what to inherit. It refuses to write any of the changes rather than apply them incompletely.

// the fix sequence

Once both problems were understood, the recovery ran in a specific order.

Step one: establish default ACL entries.

Recursive apply had to work before anything else was possible. Establishing default ACL entries on the affected datasets was the unblock.

TrueNAS SCALE ships preset ACL configurations that include both access and default entries. The RESTRICTED preset is the conservative choice — root and the dataset owner get full control, everything else is denied by default. Applying it to the dataset writes both access entries and default entries in the correct form, then recursive apply works.

Step two: recreate the SMB user with the intended group memberships.

The SMB user recreation was straightforward through the TrueNAS GUI. Same username, same password, same primary group. The new user got its new SID assigned automatically. Nothing about this step required anything unusual — the account creation is standard TrueNAS SCALE user management.

The systemd and middleware refresh sequence run after the reinstall to confirm the service layer was clean:

midclt call service.generate_all_units
systemctl daemon-reload
systemctl restart middlewared

Those weren't necessary for the ACL fix — the fresh ISO had already given us a clean service layer — but they're the standard TrueNAS SCALE service-layer refresh sequence, and running them confirmed no residual issues before the ACL work started.

Step three: apply corrected permissions recursively.

With default ACL entries in place from step one, and the SMB user existing with a new SID, the corrected ACL now needed to be written down through the dataset tree. The RESTRICTED preset had already established the ACL structure; adding the SMB user with full control on the dataset and applying recursively wrote the correct SID into every ACL entry through the tree.

Step four: verify from Windows.

Windows client connected to a share, authenticated with the SMB user's credentials, got in cleanly. No more "Credentials do not match." The credentials the user was typing had always been correct — they now matched an ACL entry whose SID pointed at the user actually trying to authenticate.

The ACL work took the same session as the reinstall itself. Most of the time was working through the preset and recursive apply through several affected datasets. The recursive apply itself was fast once it stopped erroring.

// what I'd make instinctive

Three things went into the runbook after this.

Snapshot the system config before major changes. TrueNAS SCALE lets you export the entire system config to a downloadable file — settings, users, ACL definitions, share configurations. Before this incident, I hadn't done it. If I'd had a config snapshot from before the SMB service layer failed, the fresh ISO reinstall would have restored not just the service layer but also the user accounts with their original SIDs. The SID mismatch problem wouldn't have existed. That single missing habit was the whole reason step two of the fix took the form it did. Config snapshots now go with every major change to the box.

Pool imports don't recreate everything the datasets need. Default ACL entries are the visible case; they're the one that surfaces an error message. But the principle is broader. When you import a pool into a fresh TrueNAS installation, you get the pool structure and the actual file data — you don't automatically get all the metadata state that the original installation had built up around the datasets. Some of that state has to be rebuilt manually. Assume it, don't be surprised by it.

Know the SID story before you need it. SIDs are how Windows actually attaches identities to permissions, and username is a display convention. Reinstalls change SIDs. Any ACL entry pointing at an old SID will silently fail to match a new user of the same name. The "Credentials do not match" error is misleading because the credentials aren't the problem. The identifier attached to the credentials is.

None of these are theoretical. Every one of them was learned by doing the work slightly wrong the first time.

// closing

This piece is about the layer between storage and client — where user identity meets file permissions, where authentication decisions happen. When that layer breaks, the storage is still fine, the network is still fine, and the client is confused. The SMB failure category is different from the ZFS failure category. Different response, different recovery, different mental model.

The next piece in this series is another storage-layer story — a ZFS pool where an entire RAIDZ1 vdev appeared to be gone, and the recovery required reading ZFS labels directly to confirm the disks were still valid pool members before touching anything. Different layer, different failure mode, different set of things that turned out not to be as broken as they looked.

If you're running TrueNAS SCALE and haven't exported a system config recently, do it now. That's the single highest-leverage thing you can take from this piece. The config snapshot is what turns a fresh ISO reinstall from a story about SID mismatches into a story about ten minutes of downtime.

A Kava validator on my existing stack — how the plan came together, and what the second chain will do differently

JJozzieTech — Thu, 30 Jul 2026 20:36:36 +0000

The plan didn't start at Kava. It started at the Cosmos Hub, because the flagship chain is where you go if you're not sure where to go.

The Cosmos Hub's active set — the top validators eligible for block rewards — has a floor. To enter it, a validator has to hold more staked ATOM than the smallest active validator does. In mid-2026 that floor sits at tens of thousands of ATOM — hundreds of thousands of dollars at current prices. That's an infrastructure business, not a homelab experiment. It's the size of stake where you're raising capital before you're writing a .toml file.

I don't have that capital, and I wasn't going to raise it to prove a point about self-stake. So the plan reshaped itself. Instead of validating on the flagship, target a smaller Cosmos appchain — one where a modest self-stake actually gets into the active set, but the chain is real enough that the validator work isn't performative. Kava won selection as chain #1 on that criterion. Real DeFi product with actual users, Binance-listed, healthy market cap, an active set that was reachable for a modest self-stake at the time I entered. Enough consequence to make the work matter; enough reachability to make the work possible.

Every validator writeup I've ever read assumes you got past this decision on money and skips to the software. This one starts here because for most people evaluating whether to run a validator, this is where the plan actually ends. I want the piece where it doesn't.

One thing the software will not forgive: the same cryptographic key that earns the validator's rewards will, if it ever double-signs a block, get the operator permanently and irrecoverably slashed. Not a warning. Not a temporary jail. The stake and any bonded delegator stake gets burned by the chain itself. Every design decision in what follows is downstream of that constraint.

// the hardware happened to have the right shape

The T430 wasn't acquired for this. It was already in the rack, sitting mostly idle, repurposed from a Filecoin project that didn't pan out. 512 GB of ECC on the box total, most of it committed to what had been the Filecoin VM, and the validator was going to draw from the balance. Because the RAM was there, the sizing wasn't a scarcity negotiation. There was room to be comfortable rather than minimal.

The validator VM got 8 vCPU and 48 GiB. Kava's documented minimums are lower than that. The provision deliberately sat above the documented floor rather than at it, because the downside of under-provisioning a signing node is asymmetric. A resource-starved node misses blocks. Missed blocks eventually jail the validator. Jailed validators lose bonded stake. There is no upside to a lean signing node worth the risk of a starved one, so it got comfortable margin.

The 48 GiB is a fixed allocation. Balloon disabled. Deliberate: the validator's RAM is reserved, not dynamically reclaimable, so memory contention with any other VM on the same host can't starve it. That decision followed the same pattern the Filecoin VM used, which had proven itself.

Storage: a 3.5 TB SSD, passed through to the VM as a dedicated device, mounted at /data/cosmos. Not carved from the shared LVM pool that holds VM OS disks. That separation was an explicit design choice — OS-disk failure and chain-data failure are two independent things that shouldn't share a failure mode. If the OS disk dies, the chain data survives. If the chain data corrupts, the OS is untouched. Two failures stay two failures.

The IOPS discipline was verify-before-trust. Before the node was trusted with signing, the disk was benchmarked: fsync p99 around 734 microseconds, roughly 5,353 IOPS on the workload that mattered. Adequate for the validator's write pattern, confirmed by measurement rather than assumed by spec-sheet reading. The chain is currently running lean on that volume — around 48 GB used of 3.5 TB available — so the storage layer is under no near-term pressure.

That's the hardware. Nothing about it was a targeted purchase for validating. The box had the right shape because the rack had the right box, and the decisions that made the box right for the previous project turned out to make it right for this one too.

// the snapshot that wouldn't replay

The first real war story landed at the snapshot restore step, before signing had ever happened.

Full sync from genesis wasn't the option — the chain history is large enough that starting from block zero would have run the node for days before it caught up. Snapshot restore is the standard alternative: download a recent state snapshot from a public provider, extract it to the chain data directory, start the node from there. Fifteen minutes of work, then the node picks up from a recent height and catches to tip.

The first snapshot source I tried, Polkachu, produced state that wouldn't replay cleanly. The node started, entered replay, and choked partway through with IAVL state that wouldn't reconcile under standard pruning parameters. Three attempts. Each one an evening. Each one ending the same way.

The fix was switching to a different snapshot source — NodeStake — and the replay went clean the first time. Nothing wrong with Polkachu as a provider; the specific snapshot that day just had state my node couldn't reconcile.

What came out of the pain became doctrine: verify the canonical AppHash from a public RPC before committing to a long snapshot mount. Fifteen seconds of check. If the snapshot's AppHash doesn't match what the chain reports at that height, the snapshot is bad — save yourself the hours. That check now runs before any snapshot restore on any chain in the pipeline. It's a five-line addition to a preflight script that would have saved me three evenings on this build.

// Horcrux 2-of-3: the signing architecture

Threshold signing is where the double-sign constraint gets solved.

Horcrux takes the validator's private key and splits it mathematically into shares. Each share, on its own, is useless — it can't sign anything. To produce a valid signature, a configurable threshold of shares must cooperate. The private key never exists as a whole thing on any machine; it only exists as an assembled signature at the moment it signs. There is no file on any host that, if stolen, gives an attacker a working key.

The topology choice is 2-of-3: three cosigners, any two of which are enough to sign. That specific shape is the homelab-scale sweet spot. It buys two properties simultaneously.

Fault tolerance. Any one cosigner can go down — for maintenance, a crash, a host failure — and signing continues on the remaining two. No single machine is a single point of failure for the whole validator.

No single point of compromise. No one machine ever holds a usable key. An attacker who takes one cosigner gets one share, which is worthless without a second. The double-sign threat is neutralised at the architectural level.

Alternative topologies each fail one of those tests. 2-of-2 has no fault tolerance — lose one, signing stops. 3-of-5 buys more redundancy than a homelab cluster with a handful of hosts needs while widening the surface area an attacker has to work through. For a small cluster with a no-slashing-tolerance constraint, 2-of-3 is the minimum topology that gives both properties. That's the honest reason it's the standard homelab-scale answer.

Physical distribution matters as much as the math. The three cosigners run on pro1, pro2, and pro4. The validator's full node runs on pro5. The standby node runs on pro3. No host runs both a cosigner share and a signing node. If any one physical host dies, you lose at most one share — not one share and the node together. Correlated failures at the host level get decorrelated by placement.

The honest nuance: whether every original placement decision was made purely for that reason or partly reflected which hosts had capacity, I can't fully reconstruct. What I can say is that the principle is deliberate — cosigner hosts should be separate from node hosts — and the same principle got applied more explicitly when the second chain (Akash) came online later. It reads as doctrine, and it functions as doctrine, even if the Kava-era placement was principle plus fit rather than principle alone.

The Kava-specific setup went cleanly. The shares distributed on the first attempt, the cosigners came up in the intended order, and the Horcrux Raft cluster formed quorum without incident. The hard-won signing lessons — the startup-ordering rule, the empty-placeholder-key-file gotcha, the strict host-separation rule — surfaced later on the Akash build. Those live in a future piece, not this one.

// the last check before bonding

Before the validator was ever bonded to the network, the check that mattered most was the one that proved Horcrux was actually doing the signing.

The mechanics: Kava's validator identity is normally represented by a file called priv_validator_key.json on the full node. It contains the raw private key the node uses to sign blocks. When Horcrux is doing the signing instead, that file must not be present on the node — otherwise the node would sign locally with its own key, which defeats the entire threshold-signing architecture and reintroduces the single-key-compromise risk.

The discipline is this: archive priv_validator_key.json before going live. Confirm the node is signing through Horcrux with no local key present. Only then bond stake to the validator. Only after signing-under-bond is empirically confirmed — real signed blocks recorded on the chain, produced by the threshold-signing cluster with no local key on the node — is the archived key destroyed.

The reason for that specific ordering is asymmetric downside again. If Horcrux fails silently and the node is still capable of falling back to a local key, the double-sign protection is theoretical rather than real, and the operator won't find out until it matters. If the local key is confirmed absent and the node signs anyway, the protection is real. If the local key is confirmed absent and the node stops signing, that's a fast, safe, revertible failure — you archive-restore the key, diagnose Horcrux, try again. Nobody's stake is at risk during the check.

That check is what "the double-sign will get you permanently slashed" reduces to, operationally. Everything before it is preparation. Everything after it is delegator-visible.

// first signed block, first delegator

The moment the validator was bonded and started signing, I didn't notice.

That's how it's supposed to go. When you've verified everything before going live — the storage, the sizing, the snapshot, the threshold-signing, the local-key discipline — the moment of going live is a non-event. The chain assigns a block. The validator signs it. Then another. Then another. There is no cinematic moment. There is a voting_power field on the node's /status endpoint that reads as a nonzero number instead of zero, and there are heights ticking upward in a monitoring panel, and the operator does not notice because nothing has broken.

That's success. Cinematic first-signed-block moments belong to validators that went live without checking. The absence of drama is what the pre-flight work was for.

The emotional beat, when it did arrive, was three days later. A stranger — someone I had no connection to, no relationship with, no reason to expect — looked at the validator, made an evaluation, and delegated stake to it. First independent delegator. Two total on the validator now, mine plus this one. That's the moment the whole thing stopped being an experiment and became a service someone was using.

Here's the current state, as of writing:

D+28 operational. Uptime 100%. Window block miss: 0 out of 10,000. Two delegators. That's not brag material — it's the operational state, the same way zpool status in the ZFS recovery piece was the state. Numbers on a public chain, verifiable by anyone. The point isn't the numbers. It's that this is what the pre-flight work produces.

// building the safety net: standby + drilled failover

The single node was the launch state. The current state is different.

There's now a standby node on pro3 — a second full node, kept synced to chain tip, ready to take over signing if the primary goes down. Same chain data, same signing configuration, same Horcrux endpoints. The switch from primary to standby is scripted — one script triggered by the operator with a confirmation prompt, another script to fail back to primary once it's healthy again.

The manual failover benchmark, from an earlier phase before automation, was 12 minutes 55 seconds from primary-down to signing-resumed. Human-driven. Every step verified by hand. Correct, but slow.

The scripted failover, drilled against a real deliberately-triggered outage — not a simulation, an actual systemctl stop kava on the primary — clocked at 1 minute 49.5 seconds from primary-down to signing-resumed. 37 blocks missed against the chain's 10,000-block jailing window. Roughly 0.4% of the jailing buffer consumed for a real failover event. Big enough to matter; small enough to survive.

The interesting part of the drill wasn't the failover script — it was the failback. The failback script's first dry run — the one designed to bring primary back into service after standby had taken over — surfaced two real bugs that had never been noticed. A missing privval-blank step, which would have hung real incidents to the 30-minute ceiling every time. A grep -c || echo 0 double-print anti-pattern that produced false WARNING messages downstream. Both were root-caused, fixed, and re-drilled to a clean end-to-end pass before the script was trusted with real operations. That work is exactly what a drill is for — finding bugs in the safety net before the safety net is what stands between you and being jailed.

One additional operational lesson surfaced during the drill: Horcrux's own systemctl ActiveState can report active before its Raft cluster has actually reformed quorum well enough to co-sign. Same category of "healthy-looking status ≠ real capability" issue as the chain's own voting_power field can produce, now confirmed to extend to the signing layer itself. The failback script now waits for genuine signed events, not for ActiveState=active, before declaring signing resumed.

The full topology and drill results live on the validator page. That page has the numbers; this piece has the shape.

// the first governance vote

Three weeks in, the first governance vote came up. Kava proposal 220 — recover the IBC light client for the Kava-Secret channel. Vote was YES.

The operational shape of casting a vote as a validator is different from casting one as a delegator. As a delegator, you click through a wallet UI. As a validator, you sign a governance transaction from the operator account — not the signing key, the operator key — using a discipline the validator page documents as "brief-import." The operator key is not resident on any always-on machine. It gets imported briefly for the vote, the vote transaction is signed and broadcast, and the key is removed again. Same principle as the signing-key discipline, applied to governance: the sensitive credential is never around when it doesn't need to be.

The substantive reasoning on the vote — why yes, what the recurrence of this fix pattern surfaces about relayer-set health on the Kava-Secret channel, why the meta-governance abstain vote is defensible even if it's not what I voted — lives on the on-site governance record. This piece is about the operational experience of casting the vote, not the merits of the proposal. That's a genuine split of concerns: the validator page and the governance record are the accountability artefacts, this piece is the story.

Cross-link to the governance record for the substance: /validators/kava/governance/220/.

// what would have been day-one doctrine

Some of the war stories in this piece became doctrine that got applied to the second chain. If I were starting Kava again from scratch, this is what would be day-one design rather than lesson-learned:

Startup ordering. Horcrux cosigners must be up and connected before the validator node starts its signing daemon. There's a roughly 3-second privval handshake timeout that will crash-loop the daemon if it opens its listener before a cosigner is ready to respond. Every operational script now has cosigner-side up before node-side up as a hard sequence.

The empty-placeholder-key gotcha. Some chain forks panic on startup if the priv_validator_key.json file is present but empty, rather than if it's absent entirely. The safe posture is to remove the file, not zero it. Deleted, not blanked.

AppHash verification before snapshot mount. The Polkachu-snapshot lesson from earlier. Verify the canonical AppHash from a public RPC before starting the long extraction. Fifteen seconds of check, potentially days of saved rework.

Cosigner and node physical separation as strict rule. Not "we should keep them separate where possible" — a hard rule that a cosigner share must never live on the same host as a signing node. Applied to Akash without exception. Would have been applied to Kava the same way if I'd had the discipline at day one.

Drilled failover, not planned failover. A failover script that has never been run against a real outage is theatre. HA4.5's discovery of two real bugs in a script that looked correct is the general case, not the exception. Drill the failover — with a real systemctl stop, not a simulation — before trusting it with a real incident.

None of these are theoretical. Every one of them was learned by doing the work slightly wrong the first time, and none of them cost real slashing because the safety architecture caught the mistakes before they compounded. That's what the safety architecture is for. But knowing what would have been day-one doctrine is the compounding gain — the second chain benefits from every one of these lessons without having to re-learn them.

// closing

Kava is chain #1. Twenty-eight days operational, 100% uptime, two delegators, one governance vote cast. The build wasn't dramatic. The delegator was.

The next chain is Akash. This time the shares distribution is day-one doctrine, not lessons learned later. The startup ordering is day-one doctrine, not a crash loop caught in staging. The snapshot AppHash check runs before the extraction, not after the third failed replay. The failover script gets drilled against a real outage before the validator ever bonds, not weeks after.

The homelab-first framing lands here: the validator playbook doesn't come from cloud infrastructure or a dedicated hosting provider. It comes from the rack, running alongside the rest of the workloads, doing the drilled operational work that most delegators trust cloud providers to handle. Whether that's the right posture for delegators is a conversation for someone else's article. Whether it's a defensible posture for an operator — one where the same person who wrote the failover script has run it, on real hardware, against a real outage — is what the whole series is trying to demonstrate.

Chain #1 signed its first block without ceremony. Chain #2 will sign its first block knowing what the first chain took a month to teach.

Two TrueNAS servers, one rack — the CORE-and-SCALE split I never designed

JJozzieTech — Thu, 23 Jul 2026 19:39:12 +0000

Two TrueNAS servers. Same rack. One's on CORE, one's on SCALE.

If someone asked me why, the honest answer isn't a design decision. It's a series of arrivals — one workload trigger at a time, each landing on whichever server had the shape to accept it, until the current arrangement crystallised into "how it is." Years of small choices, no whiteboard, no reorg since.

The segmentation piece covered a design I made deliberately. The OPNsense piece covered a build I saw through carefully. This piece covers a deployment that neither framing fits — because I didn't design it clean, and I didn't build it as one project. I ended up with it.

That distinction matters, because "designed," "built," and "ended up with" are three different things at homelab scale. Public writeups tend to blur them into one narrative — here's what I set out to do, here's what I ended up with, and the middle is a series of clean decisions that led logically from one to the other. Reality is messier. Workloads pivot. Constraints move. Servers get added because the workload you were adopting required them. Pools get shaped by an HBA ceiling you don't have anymore. Personal storage lands wherever there's room. And the rack ends up looking like the shape of a decade, not the shape of a design.

That's the story of my TrueNAS deployment. This piece walks through it — CORE first, SCALE later, the workload split as it stands, what actually shaped it, and the consolidation question I keep not answering.

// CORE came first

CORE went in as the storage layer for XCH — Chia — plots.

At the time, XCH needed one thing above all others: quantity of storage. Plot files are around 100 GB each and you want as many as your hardware can accept. That shaped every early decision about the box.

The R630 was the choice — a Dell 2U server, dual Xeon, plenty of ECC memory, and the important bit at the back: an external SAS HBA feeding a JBOD full of drives. The R630's own drive bays weren't enough; the JBOD was the whole point of the build.

The HBA was a SAS 6 Gbps card, and it had a drive-count ceiling I no longer remember precisely. What I remember is that it had one, and it was low enough to shape how pools got laid out. Rather than one large pool with vdevs stacked into it, I ended up with many smaller pools, each sized to what a single vdev could hold under the HBA's ceiling. That accretion of small pools is still visible in the dashboard today — vdev counts, disk-per-pool counts, all shaped by a constraint that stopped mattering when I sold most of the drives.

I ran a lot of 3 TB and 4 TB HDDs in that era. Most have since been sold as XCH scaled down. What CORE runs today is smaller than what it was built for: XCH still, and MMX — a Chia fork with its own harvester. The server is real hardware, the storage is real ZFS, and both are under-utilised relative to their original scope.

// SCALE came in with Storj

SCALE arrived when Storj replaced XCH as the direction of growth.

Storj is a distributed storage network — user data is chopped up, encrypted, and spread across independent nodes worldwide. My server hosts pieces of other people's data and serves them back on request. Different operational shape from XCH: instead of quantity of static plot files, it's active read/write traffic against smaller data volumes.

The relevant fact for this piece: Storj had a first-class TrueNAS app for SCALE, and none for CORE. Installing Storj natively meant SCALE. Running Storj on the R630 would have meant a manual container deployment, or a Docker VM alongside CORE, or something else that would carry more operational burden than clicking Install on the SCALE app catalog.

So I added SCALE. Not because I preferred it as a platform. Because the workload chose the platform.

SCALE went onto the Dell 740xd — a newer chassis than the R630, running SCALE as a VM on Proxmox on that machine. That let SCALE coexist with other workloads on the same physical server without turning it into a dedicated NAS. Storj lives there. MMX has some workload there too, alongside CORE's harvester. And over time, personal storage — media files, personal NAS shares — landed on SCALE because that's where the newer capacity was, and because SCALE's app catalog was better developed for the shape of workloads that came later.

CORE was chosen for XCH. SCALE was added for Storj. Everything else came in based on whichever server had space and shape at the time it needed to land.

// the split as it stands

Here's the current arrangement in the shortest form.

CORE (R630): XCH farming, MMX farming. Storage is JBOD-backed, structured as multiple small pools shaped by the historical HBA ceiling. The server has more capacity than the workloads currently need.

SCALE (740xd, as a Proxmox VM): Storj, some MMX, personal NAS, media files. Multiple pools, mixed drive shapes, one larger pool separating personal storage from DePIN workload storage.

No replication between the two servers. No cross-platform snapshot strategy today. Backups on both are on the horizon as part of a separate Proxmox Backup Server project — the natural place for that to land is one PBS instance handling both TrueNAS deployments plus everything else in the rack, rather than each TrueNAS doing its own thing.

The two servers don't talk to each other beyond being on the same network. They're not a cluster, they're not a pair. They're two independent deployments that happen to share a rack.

// what's shaped this beyond design

Four things shaped the current arrangement that weren't design choices.

The HBA drive-count ceiling. Real constraint in the early years. Made "many small pools" the shape rather than "one large pool with stacked vdevs." That shape persisted after the constraint stopped mattering — because reshaping ZFS pools is destructive, and there was never a moment worth spending the migration cost.

The XCH-to-alternatives pivot. When XCH scaled down as a workload, the drives went with it. What was left was capacity, not deliberate architecture. If XCH had stayed the primary workload, I probably wouldn't have added SCALE at all — the R630 alone was doing what needed to be done.

Storj's app requiring SCALE. This is the specific technical trigger. Storj's TrueNAS integration was on SCALE. If Storj had shipped a CORE plugin, or if I'd been willing to run Storj in a manually-managed container on CORE, SCALE might never have come in. One workload's platform choice became the reason for a second TrueNAS deployment.

Accretion of what came later. Personal storage, media files, the second-order workloads — they landed on SCALE not because SCALE was the right platform for them, but because SCALE was newer and had capacity. That's not architecture. That's opportunistic filling of empty space.

The pattern that connects all four: each specific arrival made sense on its own; the aggregate is a shape nobody would have designed from a blank sheet.

That's not a failure. It's what actually happens when you run infrastructure for years without a clean-slate migration event.

// the consolidation question

The natural next question: would I consolidate?

The criterion I'd apply is hardware footprint. If consolidation reduced the number of physical servers running TrueNAS in the rack, it would be worth doing. If it just moved workloads around without changing the physical footprint, it wouldn't.

That criterion doesn't currently trigger. Here's why.

The R630 is dedicated to CORE. Consolidating CORE's workloads onto SCALE would free the R630 — real hardware footprint reduction. But it would require migrating XCH and MMX plot data to SCALE, which is terabytes of data and a maintenance window I don't have. Both are non-trivial.

The 740xd runs SCALE as a VM alongside other workloads. Consolidating SCALE's workloads back onto CORE would free the SCALE VM but not any physical hardware — the 740xd would still be running other things. So the consolidation gain there is smaller.

Both servers currently run stable. No degradation pressure, no failing hardware, no capacity ceiling that forces the question. The forcing function isn't present.

PBS project is the natural precursor. Before consolidating anything, I want backup and snapshot posture solid across both platforms. That's the Proxmox Backup Server project sitting alongside this one on the roadmap. Consolidating without settled backup posture would be a much bigger risk than consolidating with it in place.

So the status is: not planned, not off the table. If the R630 develops a fault that would require investment to fix, that's the moment consolidation becomes real. Until then, both run.

The under-utilised capacity is what makes consolidation worth thinking about. The migration blast radius and the missing backup posture are what make it worth deferring.

Both those things are true simultaneously, and that's fine. Operator-honest posture doesn't require every decision to be resolved. Some decisions are open by design.

// closing

Two TrueNAS servers in one rack because history shaped it that way. CORE for XCH first, SCALE added because Storj required it, workloads accreted onto both from there.

The coming quarter doesn't change either of those facts. What changes is the PBS project settling backup and snapshot posture on both platforms — the piece that has to happen before consolidation is even a coherent question. After that, a hardware event, a workload consolidation, or a maintenance window I don't have yet.

Until then, both run.

If you're running a homelab TrueNAS today and wondering whether to migrate from CORE to SCALE — or the reverse — the answer isn't in a platform comparison. It's in your workload stack. Whether the apps you run natively on one but not the other justify the migration cost, and whether your backup posture is ready to make the migration safe. If either answer is no, the honest posture is to leave it where it is.

Sometimes the operator-grade decision is to leave things the way you found them and put the migration budget elsewhere. This is one of those decisions. Both TrueNAS servers run, and probably will for some time.

OPNsense on bare metal — 18 months as the only perimeter

JJozzieTech — Thu, 16 Jul 2026 20:32:03 +0000

For most of the homelab's life, there wasn't a firewall.

The ISP router did what ISP routers do — NAT out, block unsolicited inbound, don't ask hard questions. That's not zero security. It's just not intent-driven security. Which was fine, because the rack ran things I wrote for myself, and nobody on the internet was paying attention to my home connection.

Then I stood up an Akash provider.

The segmentation piece walked through what changed inside the rack when Akash landed — VLANs, ACLs, east-west policy. This piece is the parallel story of what changed at the edge. OPNsense on a dedicated box, ISP terminating on the WAN interface, all internal traffic on a single trunked LAN interface, running for 18 months as the only thing between the rack and the internet.

I'll cover the origin, the hardware, the split of labour between OPNsense and the Nexus 3548P, the four war stories that taught me the platform, and the four things I still haven't done that I know I should.

The last section is the point of the piece. Retrospectives that only list what worked are marketing. Retrospectives that admit what's still on the list are useful.

// no firewall, then one firewall

Before Akash, the perimeter was the ISP router.

Not zero security. Consumer-grade routers do block unsolicited inbound by default — that's been true of every ISP-supplied box shipped in the last twenty years. But the operative word is default. There was no policy. The perimeter existed because the ISP router existed. Anything more specific — logging, per-service inbound rules, outbound restriction, awareness of what was actually crossing — didn't exist because I hadn't asked for it.

That worked for a rack running things I wrote for me, running when I was watching. When Akash landed, the calculus changed. The Akash provider accepts inbound tenant deployments at an ingress that is by design publicly discoverable on-chain. Trusting the ISP router to be the only thing between that ingress and my workstation stopped being defensible once the provider was live and taking inbound deployments.

The natural sequence, once I was building for that: segment first, then police. The segmentation piece covered the segment step. This is the police step.

OPNsense went in on a dedicated box. ISP terminated on its WAN interface. Every internal VLAN — management, workload VLANs, storage VLAN — trunked into a single LAN interface. The rack now had two policy-driven layers separating it from the internet: the segmentation inside, and OPNsense at the edge. Neither trusts the other by default.

// why bare metal, not a VM

The obvious cheap answer would have been an OPNsense VM on Proxmox. It would have run fine. I didn't do that. Three reasons, in the order I actually thought about them.

Compute belongs to workloads. The Proxmox hosts exist to run DePIN infrastructure — Akash, storage, harvesters, validators. Every VM slot I give to a perimeter firewall is a slot I don't give to something earning. Bare metal at the edge means the servers stay full of what they're for.

The perimeter is a different animal from the compute. The firewall shouldn't share a fate with the hypervisor. If Proxmox goes sideways — kernel panic, storage-driver bug, upgrade gone wrong — the perimeter should be uninvolved. Bare metal enforces that separation physically, not just conceptually.

ISP terminates here. OPNsense is the box the ISP handoff talks to. That relationship is cleaner when exactly one physical device owns it, and when that device isn't a guest on a hypervisor whose primary job is something else entirely.

The hardware, as it exists in the rack today:

model:      dell optiplex 7080 SFF
cpu:        intel core i7-10700 (8c/16t)
memory:     32 gb
storage:    SATA SSD, built-in bay
nics:       1× onboard intel i219  (WAN)
            4× intel i350 add-in card  (LAN + 3 spares)
os:         opnsense 25.7 (freebsd base)
form:       SFF, rack shelf

The i7-10700 is overkill for what OPNsense currently asks of it. This rack doesn't push enough traffic to stress an 8-core Comet Lake at the edge. But headroom is what enables the still-on-the-list items — IDS at line rate needs cores, and a CARP secondary needs a matched-pair budget to be defensible.

Between OPNsense and pfSense: open-source posture and the UI. The plugin ecosystem sealed it.

// the split of labour

The segmentation piece established the split. Sharpening it for this piece:

The Nexus 3548P handles east-west. Inter-VLAN routing at line rate on the switching ASIC. Workload-to-workload isolation lives on the Nexus, enforced by ACLs on each SVI.

OPNsense handles north-south. WAN, NAT, port-forwards, outbound NAT posture, inbound perimeter policy. Anything that crosses the boundary between the rack and the internet.

Two rulesets, two devices, two audit surfaces. Both stay simpler.

Interface layout at the OPNsense end:

WAN interface   →  ISP handoff, single link, dhcp from provider
LAN interface   →  tagged trunk to nexus, carries all internal vlans
spare ports     →  available for CARP secondary or dedicated management

One physical LAN interface is enough because the workload VLANs are tagged on the trunk. OPNsense sees each VLAN as a virtual interface on top of the physical LAN port and can write rules against each independently — inbound and outbound, per-VLAN, without the physical port count constraining what's possible.

The spare ports on the add-in card matter for what's coming. When the CARP secondary goes in, pfsync between the pair needs its own link. That's what one of the spares becomes.

// what the rack does with NAT

OPNsense defaults to Automatic outbound NAT: one masquerade rule per LAN interface, everything else inferred. That's what most homelab OPNsense builds run. It's the correct default. It also isn't what this rack runs.

This rack runs Manual outbound NAT. Every workload VLAN has an explicit outbound rule referencing a named network alias. The pattern:

alias:  workload-a-net   →   10.x.x.0/24

rule:   source workload-a-net   →   NAT to WAN interface address

Two reasons that's worth the extra config surface.

Explicit is better than inferred. When you're staring at a firewall trying to work out whether workload X can reach the internet, seeing an explicit named rule beats reasoning about Automatic-mode inference every time. The rule either exists or it doesn't. There's no "well, in Automatic mode it would generate…" middle ground.

Aliases are reusable. The same workload-a-net alias that appears in the outbound NAT rule can be referenced in inbound rules, port-forwards, floating rules — anywhere the subnet needs to appear. One object, one place to update. When a workload's subnet changes — or when it moves — one alias edit propagates everywhere.

The trade-off: Manual mode means new workloads don't get NAT for free. Adding a workload means adding an alias, adding a rule, and applying. That friction is a feature. It means every workload on the wire had a moment where someone explicitly decided how it talks to the internet. Nothing gets to the outside by accident.

// four things OPNsense taught me

Four incidents from 18 months in. Each taught a different class of lesson.

1. The saved config that wasn't the applied config.

A static route to a workload subnet, correctly configured in OPNsense — and workloads on that subnet couldn't reach the internet. Firewall rules checked out. NAT checked out. The routing table on the L3 switch was correct. The route was in the OPNsense config screen but not in the OPNsense routing table.

Pending changes had been staged and never applied. Clicking Apply activated the route and connectivity came back immediately.

Lesson: saved is not applied. OPNsense stages changes to firewall, NAT, and routing configs, and requires an explicit Apply to activate them. The running config can silently drift from the staged one if you forget. Any troubleshooting session that starts with "but the config looks right" should also check whether pending changes are waiting.

2. The subnet OPNsense wouldn't accept as a NAT source.

Configuring outbound NAT for a workload VLAN. Enter the workload subnet as the source. OPNsense refuses: "not a valid source IP address or alias."

The subnet was valid. The rule editor just wouldn't accept it in raw form.

The fix: create a named network alias containing the subnet, then reference the alias in the rule. Alias workload-a-net containing 10.x.x.0/24, and the rule takes it without complaint.

Lesson: OPNsense treats aliases as first-class objects. Some rule fields insist on aliases even when a raw CIDR is technically valid. This isn't a bug — it's the OPNsense-idiom answer to "how do we make firewall rules maintainable at scale." Once the reflex to reach for aliases is in place, everything gets more consistent.

3. The firewall rule that wouldn't edit.

An inbound WAN rule showed up in the firewall ruleset but the edit UI wouldn't let me change it. It looked like a normal rule. It just wouldn't respond to edits.

The rule was auto-generated by a NAT port-forward. OPNsense's port-forwards have a Filter rule association setting that automatically creates the corresponding firewall rule and manages it as a dependency of the port-forward. Direct edits either don't apply or get overwritten the next time the port-forward is touched.

Fix: either edit the port-forward's own rule fields (which flow through to the generated firewall rule), or set the association to "None" and manage the firewall rule separately.

Lesson: know which rules OPNsense controls and which you control. Not every rule in the ruleset is an independent object. Assuming otherwise costs you an afternoon.

4. The package update that wouldn't complete.

A routine package and firmware update ran into a repository / package-consistency issue and stopped mid-upgrade. Recovering the box required diagnosing the state of the pending upgrade rather than just retrying. Endpoint landed at OPNsense 25.7.3_7 eventually.

The firewall stayed on-line during the diagnosis. That was luck. I hadn't taken a config backup before the update, and I hadn't confirmed console access. If the update had left the box in an unusable state, I would have been rebuilding OPNsense from memory.

Lesson: before touching updates, capture a config backup and confirm console access. Both take five minutes. Neither had happened. Both are habits now.

// what I still haven't done

Four gaps. In the order I'd actually tackle them if I started this weekend.

1. Backups first.

The single most embarrassing gap. OPNsense's config is an XML export that can be committed to Git, uploaded to cloud storage, or scheduled to a network share via the built-in backup module. None of that has happened here.

Backups are first not because they're the most urgent to have, but because they're the cheapest to get. Ten minutes of config setup and I have a restorable state. Every subsequent gap-closing effort is safer once this is in place. The next OPNsense update — and it'll happen — will be less nerve-wracking with a known-good config sitting in a repo somewhere.

2. HA second.

Would I make the same bare-metal call again? Absolutely. Would I have planned CARP into the initial build from day one? Also absolutely.

Retrofitting HA to a running perimeter is harder than designing it in. CARP virtual IPs need to be planned around the WAN handoff. pfsync between peers needs a dedicated link. Both are easier to reason about with a matched pair of boxes than with an active production box and a second box being brought into a pair. The next OptiPlex is the plan. It's a project, not a config change.

3. Logging third.

OPNsense's logs currently live on the box and roll over. That's fine for "check the last hour's blocked traffic." It isn't fine for anything else. The natural target is either a Loki/Grafana stack in the rack or Graylog on a dedicated VM.

Logging goes before IDS/IPS because IDS/IPS generates the logs. Having nowhere for them to land defeats the point.

4. IDS/IPS last.

Suricata on the LAN interface, ET Open ruleset, IDS mode first (observation) before IPS mode (enforcement). Deliberately last, because the false-positive load is real, the tuning effort is real, and the payoff without steps 1–3 is limited. IDS/IPS without a config backup, without HA, and without a place to send logs would just add fragility instead of catching it.

The order is deliberate. The temptation with lists like this is to lead with the exciting-sounding gap. Backups aren't exciting. They're the load-bearing gap that everything else depends on.

// closing

The perimeter has held for 18 months. Zero incidents I've traced to the firewall itself. Two of the four war stories above (the un-applied config, the un-editable rule) taught me how the UI thinks. Two (the alias requirement, the package update) taught me how the platform thinks. All four made me more useful, not less.

The next 18 months aren't going to be about new capability. They're going to be about closing the four gaps. A backup piece. A CARP-HA piece. A logging-to-something-real piece. Eventually an IDS/IPS piece, once the first three are in place to catch what it finds.

If you're running a homelab without a firewall today because the ISP router is fine, you're where I was two years ago. The trigger to move isn't a security incident — those don't announce themselves. The trigger is any workload you'd rather not have sharing a box with your workstation, or any inbound traffic you can't defend by hand-waving at NAT.

If you already have OPNsense running: check whether your config is backed up. That's the single highest-leverage thing you can do this week.

How I got a suspended TrueNAS pool back online without replacing a single disk

JJozzieTech — Thu, 09 Jul 2026 02:54:20 +0000

The rack had just come back online after an outage. Most things looked fine — VMs booted, the network reconverged, monitoring caught up on the missed intervals. Then I ran zpool status on one of the TrueNAS SCALE nodes and saw output I hadn't seen in production before.

An 11-disk RAIDZ1 vdev. Multiple drives marked as faulted. One member listed as removed. The pool itself in SUSPENDED state.

The reflex, in that moment, is to start replacing disks. That reflex is wrong — at least as the first move. This piece is how I got the pool back online without touching a single drive, and what I learned about how ZFS reports failure states along the way.

The pool in question is on my TrueNAS SCALE machine, which runs alongside a separate CORE box on a different chassis. The recovery pattern is the same on either — this isn't SCALE-specific ZFS behaviour — but the incident happened on SCALE, so that's where the console output in this piece comes from. For privacy, I'll refer to the affected pool as MediaPool-A.

// what the console showed

The output was the classic suspended-pool warning:

  state: SUSPENDED
 status: One or more devices are faulted in response to IO failures.
 action: Make sure the affected devices are connected, then run 'zpool clear'.

Then the vdev listing. The RAIDZ1 topology looked much worse than it turned out to be — most of the 11 disks were showing as faulted, and one was marked as removed. If you'd shown me that output out of context and asked me to guess what happened, I'd have said "shelf died" or "backplane failure." The vdev looked shot.

But there was a shape to the failure that mattered. Multiple disks going down at the same time is exactly what you'd expect if something they shared had gone down — an HBA path, a SAS cable, a backplane, a controller — rather than the disks themselves failing individually. Simultaneous multi-disk failure is one of those things that sounds catastrophic and usually isn't. Independent failures don't cluster in time. Shared-path failures do.

That's the diagnostic reframe the piece is built around. Before you touch a disk, ask what those disks have in common that isn't the disks.

// the reflex to resist

The most important part of ZFS recovery is what you don't do in the first ten minutes. Here's the list, in order of danger:

Don't add disks to the pool. Whatever's shown as unassigned in the TrueNAS GUI may not actually be unused. If the pool is suspended and the disks are still recognised by the OS, they're still valid pool members even if the GUI is being coy about it. Adding them to a new vdev structure is the fastest way to permanently lose the pool.
Don't wipe any disk. Same reasoning. If a disk still has valid ZFS labels for this pool, wiping it is destructive.
Don't replace any disk. Not until you've confirmed there's a disk that actually needs replacing. See earlier point on multi-disk-fault patterns.
Don't export or destroy the pool. zpool destroy is exactly what it sounds like. zpool export is safer but shouldn't be done while the pool is in an unstable state.
Don't start a scrub while the pool is suspended. A scrub is not the recovery mechanism when the pool has been suspended due to I/O access problems. The pool needs to be brought back into a stable online state first. Scrubbing before that either does nothing useful or, worse, hammers the disks while they're on an unreliable path.

That last one catches people. Scrubs are the go-to "is everything OK?" tool in ZFS-land, and the muscle memory when something looks wrong is to run one. Muscle memory is wrong here. A scrub is validation, not recovery. It belongs at the end of the recovery process, not the start.

// step one: are the disks actually there?

Before running anything against the pool, I checked whether the operating system could see the disks. This is the cheapest possible diagnostic and it usually tells you which world you're in.

smartctl --scan

The disks were visible. All of them. That was the first good sign — it meant the OS could see the hardware, which meant whatever had happened during the outage had resolved at the hardware/path level, which meant the pool was recoverable in software without touching any physical component.

Then I checked kernel logs for signs of ongoing HBA or SAS trouble:

dmesg -T | grep -iE "sas|scsi|mpt|mps|mpr|reset|timeout|failed|medium error|sense|I/O error|device offline|link"

The kernel log showed events from the outage itself — expected, since the disks had briefly gone away and come back — but nothing recent. No active storms of SAS resets. No timeouts still firing. No disks going offline. The path was stable now, whatever had happened during the outage.

That was the second good sign, and it was the one that gave me confidence to move to the recovery step. If the kernel had still been throwing SAS resets or timeouts in the current second, I would have stopped, escalated to the physical layer — cables, HBA, PSU — and only come back to ZFS after the path was clean. Running zpool clear against a still-unstable path just re-triggers the fault, and does it at higher urgency this time.

// zpool clear, safely

Once the two "is the path stable?" checks came back clean, the recovery command was the one ZFS itself had already suggested:

zpool clear MediaPool-A
zpool status -v MediaPool-A

The pool returned to an ONLINE state. One disk briefly showed a small number of checksum errors on the first status check after the clear — expected, because the outage had generated some genuinely bad reads that ZFS had queued up. A second zpool clear zeroed the counters, and the follow-up status came back clean:

MediaPool-A = ONLINE
all disks = ONLINE
READ / WRITE / CKSUM = 0
errors = No known data errors

No files reported as corrupted. No permanent errors. The pool was back.

The recovery itself was two commands. The rest of the work was in the "is the path actually stable?" checks before them — cheap to run, and the difference between a clean clear and re-triggering the fault at a moment when the pool was already suspended.

// what actually happened

The most likely cause was a temporary I/O path disruption during the outage. Something in the shared path — HBA, cable, expander, backplane, or the disks all briefly losing power at slightly different times as the rack came back — presented as multiple simultaneous device failures to ZFS. ZFS did exactly what it's designed to do: suspended the pool to protect the data from being written to devices it wasn't sure were healthy.

Once the path stabilised, the disks were visible again. zpool clear told ZFS "the fault condition is resolved, retry," and ZFS did. Because the disks and their data were still intact underneath, the retry succeeded, and the pool came back with no permanent state loss.

The three ZFS terms that mattered here are worth stating precisely, because their intuitive meanings are misleading:

FAULTED does not always mean a dead disk. It means ZFS has decided this device is unreliable enough to stop trusting for the moment, based on the errors it's seen. A device can be FAULTED for reasons that have nothing to do with the disk itself.
REMOVED does not always mean physically removed forever. It means the device that was here at some point isn't visible right now. It can be transient.
SUSPENDED does not always mean the pool is lost. It means ZFS has stopped I/O on the pool to protect it, pending resolution of whatever fault condition triggered the suspension.

Context is what tells you which of those interpretations applies. Multiple simultaneous FAULTED devices, immediately after a rack outage, on drives you can still see with smartctl and don't have active kernel errors on, points at "transient path disruption during the outage" and away from "disks died." A single FAULTED device weeks after normal operation points the other way, and the response is completely different.

// what I'd make instinctive

Two things, and one thing I already had right.

Look at the shared layer first. When multiple disks fault at once, the useful question is not "which disks failed?" but "what do they share that could have failed?" HBA, SAS path, backplane, PSU, controller firmware. Independent failures don't cluster in time; shared-path failures do. Every panic-response-to-multi-disk-fault I've ever done has, eventually, come back to something at the shared layer. Skipping straight to the shared layer as the first diagnostic saves you the panic.

zpool clear is safe only after the path is confirmed stable. It is not a magic fix. If the underlying path is still unstable, zpool clear re-triggers the fault immediately, and the second suspension arrives faster than the first because ZFS has now seen the pattern twice. Confirming path stability first — smartctl --scan, then dmesg | grep for active errors — takes about thirty seconds and is what makes zpool clear a recovery tool instead of a way to make things worse.

The thing I had right: not scrubbing while the pool was suspended. That's an instinct worth keeping. Scrub validates a stable pool. It doesn't recover an unstable one.

// closing

This was a clean recovery because the underlying condition turned out to be transient. The pool came back with no touch to any physical disk, no wipe of any data, no rebuild of any vdev. The recovery cost was two commands and about ten minutes.

Not every ZFS recovery is that clean. The same outage that suspended MediaPool-A also took down a second pool on the same box in a different, harder way — one where an entire RAIDZ1 vdev appeared to be gone, and the recovery required reading ZFS labels directly to prove the disks were still there before running any of the commands above. That's the next piece.

If you're recovering from something like this now: check the disks are visible, check the kernel logs are quiet, then run zpool clear. In that order. Everything else you might be tempted to do can wait.

I run my homelab like a miniature data centre — here's the network design that made it possible

JJozzieTech — Thu, 02 Jul 2026 06:57:36 +0000

The homelab started flat. One /24, everything on it. My workstation, the NAS, the Proxmox host, and — over time — a growing list of workloads sharing the same broadcast domain because that was the path of least resistance.

For a while, that was fine. A homelab running one workload doesn't need segmentation any more than a house needs an office door.

Then I stood up an Akash provider.

An Akash provider is, in shape, a Kubernetes cluster that accepts inbound tenant workloads from the internet — real deployments, paying for compute, containers I didn't write landing in namespaces on my hardware. The provider itself is documented at github.com/jjozzietech/akash-provider-ops-public — this piece is about the network underneath it. The containerisation posture itself is fine. I trust the isolation model. But trust isn't a network design. And the network at that moment had the tenant workload cluster sitting on the same subnet as my workstation, my NAS, and my Proxmox management interface.

That was the moment I stopped thinking of the rack as a home network with extra boxes, and started thinking of it as a small data centre.

This piece is the network design that came out of that shift. I'll cover the layout, the rules that hold it together, and the Nexus and Proxmox configs that anchor it — with the specifics of my own deployment sanitised. It's not a step-by-step replication guide. It's the design pattern, with enough of the shape to be useful and enough restraint to not double as a recon document for my own rack.

// the original design

The flat layout looked like this:
home lan — 192.168.1.0/24

opnsense (perimeter)

cisco nexus (dumb L2 switching)

proxmox host

workload VMs (all on the same subnet)

What it got right: zero routing complexity, everything reachable from everywhere, fast to stand up. If you're running one project on a homelab, this is the correct design. Don't over-engineer it.

What stopped working, as soon as the second project landed on the rack, was that the design had no answer to three questions:

How do I write different firewall policies for different workloads when they all share a subnet?
How do I keep a compromise in workload A from reaching workload B?
How do I troubleshoot when every packet in the rack lives in the same broadcast domain?

The last one is the operational tell. When "which workload is chatty on the wire right now?" becomes a question you can't answer from a switchport counter, the design has outgrown the problem it was designed for.

The industry term for what I was worried about is east-west traffic — traffic between workloads on the same tier, as opposed to north-south traffic between workloads and the outside world. Flat L2 doesn't give you any policy handle on east-west. You just have to trust that nothing on the wire behaves badly.

That's a defensible assumption for a house LAN. It's not a defensible assumption for infrastructure that hosts tenant workloads.

// the reframe: stop thinking residential

Commercial data centres don't put every workload in the same subnet. They isolate services into dedicated network segments, each with its own routing policy, its own firewall posture, its own blast radius. When something goes wrong, the fault is bounded. When something needs a different security posture, it gets one. When a workload needs to move to different hardware or a different site, the network context comes with it.

That's the model I took. Every project on the rack gets:

Its own VLAN
Its own /24
Its own firewall policy at the routing boundary
No default trust with any other VLAN

The hypervisor stays on management. Workloads live on their own VLANs. Ops access is explicit, not implicit.

A quick note on hardware, because it matters. The switch doing this work is a Cisco Nexus 3548P — enterprise-grade L3 switching, ACLs applied inbound on the SVI, inter-VLAN routing at line rate on the switching ASIC rather than punted to a CPU. A UniFi or Mikrotik can do most of what's below, and if you're building this pattern on consumer gear, that's a reasonable place to start. What you give up is the ability to enforce policy at wire speed with the ACL evaluated in hardware. On a homelab-sized rack it's rarely the bottleneck. On a rack that hosts real workloads with real tenants, I wanted the ceiling raised.

// the architecture

Here's the layout, in the same YAML metadata style I use across the site:
management: 192.168.1.0/24 workstation, opnsense, proxmox host

vlan N (workload-a): 10.N.N.0/24 first project

vlan M (workload-b): 10.M.M.0/24 second project

vlan …: … future

Actual VLAN IDs and subnets in my deployment differ. The pattern is what matters, and I'll get to why the pattern is the way it is in the "what I'd do differently" section.

Three design rules underpin this.

The hypervisor stays on management, VMs live on workload VLANs. The Proxmox host is not a workload. It's the platform workloads run on. Putting it on the management VLAN means the host stays reachable when a workload VLAN goes sideways — backups continue, console access works, snapshots run. If the Proxmox host is on the same VLAN as a workload and that VLAN's routing breaks, you've just lost your recovery path.

One /24 per workload, gateway on the Nexus. The L3 boundary between VLANs is where policy lives. That means the Nexus SVI, not OPNsense. OPNsense's job is different, and I'll get to it.

Workload VLANs are peers, and no VLAN trusts another by default. East-west is denied. ACLs grant the exceptions. If workload A ever needs to talk to workload B, that's a deliberate rule, not a side effect of shared broadcast domain.

// what the nexus does

Three sanitised snippets, each with a sentence on intent.

VLAN definition:
vlan 10

name workload-a

Nothing exciting. The VLAN is a container; the interesting policy lives on the SVI.

SVI with gateway and inbound ACL:
interface Vlan10

description workload-a-gw

ip address 10.10.10.1/24

ip access-group acl-workload-a in

no shutdown

The ip access-group acl-workload-a in line is the important one. Inbound on the SVI means traffic entering the Nexus from the workload VLAN is what gets filtered. Any packet leaving workload A's subnet passes through that ACL before it gets routed anywhere else. That's how east-west enforcement happens at the routing boundary.

Trunk to a Proxmox node:
interface Ethernet1/3

description pve01-trunk

switchport mode trunk

switchport trunk allowed vlan 10,20,30

spanning-tree port type edge trunk

no shutdown

The switchport trunk allowed vlan list is explicit. Only VLANs I've deliberately added to the trunk can traverse it. New workload comes online → the VLAN gets added to the allow-list. Mundane, but it means the trunk isn't a wide-open pipe by default.

I leave the native VLAN unset on workload trunks. Tagging is explicit on the Proxmox side, which I'll cover shortly.

// ACLs are the actual security mechanism

VLAN segmentation isolates broadcast domains. That's it. Two VLANs on the same switch, both trunked to the same Proxmox host, can absolutely talk to each other — as soon as one of them has a route to the other. The route is provided by the SVI. Which is why the SVI is where policy has to live.

The pattern, not the rules, for a workload VLAN's ACL looks like this:
acl-workload-a:

permit management-workstation -> workload-a (ops access in)

permit workload-a -> dns, ntp (outbound essentials)

deny workload-a -> 10.0.0.0/8 (block east-west)

deny workload-a -> 172.16.0.0/12 (block east-west)

deny workload-a -> 192.168.0.0/16 (block east-west)

permit workload-a -> any (internet last)

Walking through the logic:

Rule order matters. The three RFC1918 denies sit before the permit-any-internet. Reverse them and the denies never evaluate — the permit-any matches first and every internal network is reachable. This is the single most common way I've seen this pattern implemented incorrectly.

The management workstation gets an explicit permit at the top. Without it, I lose ops access to the workload VLAN — SSH, monitoring, any control-plane traffic. With it, only that one host can reach in. Everything else that tries to reach workload A from a private range hits the RFC1918 denies further down.

DNS and NTP get their own permit lines. Cleaner than carving exceptions into the deny block. The workload needs to resolve names and know what time it is; both are explicit.

No VLAN-to-VLAN permits exist. If workload A needs to talk to workload B in the future, that'll be a deliberate exception — added, logged, reviewed. Default deny is the actual default, not aspirational.

What's not in this article: the per-host permits, the port restrictions inside the workload VLAN, the exact management-range definition, and the north-south inbound permits handled at the OPNsense boundary. The shape is the point. The full ruleset stays on the rack.

// dividing the labour: nexus vs opnsense

There's a clean split between the two devices.

The Nexus handles east-west: inter-VLAN routing and policy. Everything that stays inside the rack.

OPNsense handles north-south: WAN, NAT, port forwards, outbound NAT, anything that crosses the perimeter to or from the internet.

Two devices, two audit surfaces, two separate rulesets. Both stay simpler as a result.

The alternative — collapsing everything into OPNsense as a router-on-a-stick — is a design I see in a lot of homelab writeups, and it works. But it has two costs. First, inter-VLAN traffic becomes CPU-bound on the firewall instead of running at line rate on the switch. Second, north-south and east-west rules end up in the same ruleset, which makes both harder to reason about over time. You end up asking "does this rule apply to WAN traffic or inter-VLAN?" every time you touch the config, and the answer depends on interface assignments buried elsewhere.

Keeping the two roles physically separate — Nexus for east-west, OPNsense for north-south — is more disciplined. And on the Nexus, inter-VLAN routing doesn't touch the firewall at all, which means an OPNsense reboot doesn't interrupt workload-to-workload traffic. (It also doesn't enable workload-to-workload traffic, because the Nexus ACL denies it by default. But the point is that the firewall isn't in the east-west path either way.)

// the proxmox side

The Proxmox network config on the host is deliberately simple. One VLAN-aware Linux bridge, one LACP bond feeding it, management IP on the untagged side.

Here's the shape, sanitised — real interface names, redacted IPs:
auto lo

iface lo inet loopback
iface ens7f0 inet manual

iface ens7f1 inet manual
auto bond0

iface bond0 inet manual

bond-slaves ens7f0 ens7f1

bond-miimon 100

bond-mode 802.3ad

bond-xmit-hash-policy layer2+3
auto vmbr0

iface vmbr0 inet static

address 192.168.1.10/24

gateway 192.168.1.1

bridge-ports bond0

bridge-stp off

bridge-fd 0

bridge-vlan-aware yes

bridge-vids 2-4094

Two physical NICs (ens7f0 and ens7f1) bond together into bond0 using LACP (802.3ad) — the switch side is configured as a matching port-channel. That gives me two paths to the Nexus, active/active, with automatic failover if one link drops.

The bond feeds a single Linux bridge, vmbr0, configured as VLAN-aware. The bridge trunks every VLAN from 2 to 4094 (in practice, only the VLANs actually on the switch's allow-list traverse it). The hypervisor's management IP sits on the bridge itself on the untagged VLAN — which is the management VLAN, reachable from the workstation and the rest of the management infrastructure.

The VM side is one line of config per VM: the NIC gets a VLAN tag set in the Proxmox GUI. The VM sees a regular Ethernet interface. The bridge handles tagging on the way out to the bond, and the Nexus routes based on the tag. That's the whole story on the host.

Notably absent: a separate bridge per VLAN. That design works, but it doesn't scale — every new VLAN means a new bridge, a new set of uplinks to manage, and a growing surface of things that can drift out of sync between hosts. One VLAN-aware bridge trunked to the switch does the same job with dramatically less state to maintain.

// what I'd do differently

Two things: one process, one design. Plus the honest note on the migration itself.

Process: design the ACLs before the workload goes live, not after.

The segmentation itself I got right the first time. The ACLs I got right eventually — but I built them reactively, as workloads landed and I discovered gaps. New workload goes live → I stand up the VLAN and SVI → I permit the ops access I need to configure it → the workload comes up → I notice traffic leaking somewhere it shouldn't → I tighten the ACL. Each iteration got closer to correct. But each iteration also had a window where the ACL was more permissive than I wanted, and one of those windows was long enough to matter.

The right approach is to have a template for both directions of policy before you provision anything. The east-west deny block as a ready-to-paste chunk. The north-south permit list — which ports OPNsense will forward inbound, from where — drafted alongside the workload's own deployment plan. Apply both at the moment the VLAN is created. The workload never exists in a permissive state, even briefly. The "did I remember to tighten that rule?" question never comes up.

This is the single thing that separates the segmentation I have now from segmentation I'd have designed on day one if I could start over.

Design: I'd keep the VLAN scheme exactly as it is.

The pattern is VLAN N maps to 10.N.N.0/24. VLAN 10 lives in 10.10.10.0/24. VLAN 11 lives in 10.11.11.0/24. And so on.

It's opinionated and I'll defend it. The payoff is that troubleshooting is self-documenting. A packet capture shows a host in 10.10.10.0/24 and I know without looking that it's on VLAN 10. An ACL entry references 10.11.11.0/24 and I know without looking it's targeting VLAN 11. Every debugging session, every ACL review, every firewall rule audit is faster because the VLAN ID and the subnet are the same number.

The downside is you're locked into a /24 per VLAN, and you can't ladder easily beyond VLAN ~250 before you start running out of clean second-octet space. For a homelab, that's not a real constraint. If it ever becomes one, the failure mode is switching to a different scheme for the overflow VLANs — the existing ones don't need to change.

Migration: rolling, and it was fine — mostly.

The migration itself was rolling rather than greenfield. I had running workloads, no maintenance window, and no ability to stand up a parallel network to cut over to. So each project moved one at a time: stand up the new VLAN, ACL, and SVI on the Nexus; add the VLAN to the Proxmox trunk allow-list; retag the VM's NIC; update any OPNsense port forwards to the new IP; verify; move on to the next.

Here's the one that stung. My Storj storage node went dark for longer than I'd like to admit during the TrueNAS cutover. TrueNAS moved from the flat network to its own VLAN — new subnet, new IP. The Storj app runs as a TrueNAS CORE plugin, so it inherited the new IP automatically. What didn't inherit anything automatically was the OPNsense port-forward, which cheerfully continued sending inbound Storj traffic to TrueNAS's old address. From my side, everything looked fine — TrueNAS was up, the plugin was running, SMB shares were serving internal clients without complaint. From the Storj network's side, my node had stopped accepting inbound connections. It took a reputation dip before I noticed.

The failure mode is generic and worth naming: when a workload moves subnets, every downstream reference to its old address is now stale. Port-forwards, reverse proxies, DNS records, anything with a hardcoded IP in a config, anything scraping Prometheus on the old address. My cutover checklist was fine for the workload itself. It was not fine for everything that talked to the workload. That's the kind of gap a pre-provisioned north-south permit template — with every port and destination mapped out before the move — closes automatically. My checklist now includes an explicit "audit every reference to the old IP" step. It hasn't happened again.

// scaling: why this pays back

The marginal cost of adding workload N+1 is now:

Pick the next VLAN ID and its matching /24.
Define the VLAN and SVI on the Nexus, with the template ACL applied.
Add the VLAN to the Proxmox trunk allow-list.
Tag the VM's NIC.
Configure north-south forwarding on OPNsense if the workload takes inbound traffic.

That's a fifteen-minute job for a workload that doesn't need custom firewall policy, and maybe an hour if it does. Not a redesign. Not a rework of anything already running.

The same logical layout transfers to colocation or a commercial data centre with the physical topology changing but nothing else. The VLAN IDs, the subnets, the ACLs, and the OPNsense rules all move as-is. That was one of the original design constraints — build a homelab network that doesn't need to be rethought when the rack does — and it's the part I'm most quietly proud of.

// closing

Most homelabs don't need any of this. If you're running a Plex box and a media NAS and maybe one game server, flat L2 is the right answer. The complexity here only starts paying back when you're running multiple workloads with materially different trust postures — something you built yourself next to something you didn't, something that accepts inbound traffic from the internet next to your workstation, something in production next to something in perpetual dev.

The moment you cross that line, segmentation stops being over-engineering and starts being basic operational hygiene.

This design pattern is the substrate the rest of the rack runs on. The next few pieces in this series build on it — OPNsense as the perimeter, TrueNAS as the storage layer, and eventually the workload-specific pieces where individual services get their own writeups. All of it assumes the network underneath looks like this.

If you're running mixed workloads on a flat network today, the question isn't whether to segment. It's how many workloads you want to add before you do.