DEV Community: JJozzieTech

Two TrueNAS servers, one rack — the CORE-and-SCALE split I never designed

JJozzieTech — Thu, 23 Jul 2026 19:39:12 +0000

Two TrueNAS servers. Same rack. One's on CORE, one's on SCALE.

If someone asked me why, the honest answer isn't a design decision. It's a series of arrivals — one workload trigger at a time, each landing on whichever server had the shape to accept it, until the current arrangement crystallised into "how it is." Years of small choices, no whiteboard, no reorg since.

The segmentation piece covered a design I made deliberately. The OPNsense piece covered a build I saw through carefully. This piece covers a deployment that neither framing fits — because I didn't design it clean, and I didn't build it as one project. I ended up with it.

That distinction matters, because "designed," "built," and "ended up with" are three different things at homelab scale. Public writeups tend to blur them into one narrative — here's what I set out to do, here's what I ended up with, and the middle is a series of clean decisions that led logically from one to the other. Reality is messier. Workloads pivot. Constraints move. Servers get added because the workload you were adopting required them. Pools get shaped by an HBA ceiling you don't have anymore. Personal storage lands wherever there's room. And the rack ends up looking like the shape of a decade, not the shape of a design.

That's the story of my TrueNAS deployment. This piece walks through it — CORE first, SCALE later, the workload split as it stands, what actually shaped it, and the consolidation question I keep not answering.

// CORE came first

CORE went in as the storage layer for XCH — Chia — plots.

At the time, XCH needed one thing above all others: quantity of storage. Plot files are around 100 GB each and you want as many as your hardware can accept. That shaped every early decision about the box.

The R630 was the choice — a Dell 2U server, dual Xeon, plenty of ECC memory, and the important bit at the back: an external SAS HBA feeding a JBOD full of drives. The R630's own drive bays weren't enough; the JBOD was the whole point of the build.

The HBA was a SAS 6 Gbps card, and it had a drive-count ceiling I no longer remember precisely. What I remember is that it had one, and it was low enough to shape how pools got laid out. Rather than one large pool with vdevs stacked into it, I ended up with many smaller pools, each sized to what a single vdev could hold under the HBA's ceiling. That accretion of small pools is still visible in the dashboard today — vdev counts, disk-per-pool counts, all shaped by a constraint that stopped mattering when I sold most of the drives.

I ran a lot of 3 TB and 4 TB HDDs in that era. Most have since been sold as XCH scaled down. What CORE runs today is smaller than what it was built for: XCH still, and MMX — a Chia fork with its own harvester. The server is real hardware, the storage is real ZFS, and both are under-utilised relative to their original scope.

// SCALE came in with Storj

SCALE arrived when Storj replaced XCH as the direction of growth.

Storj is a distributed storage network — user data is chopped up, encrypted, and spread across independent nodes worldwide. My server hosts pieces of other people's data and serves them back on request. Different operational shape from XCH: instead of quantity of static plot files, it's active read/write traffic against smaller data volumes.

The relevant fact for this piece: Storj had a first-class TrueNAS app for SCALE, and none for CORE. Installing Storj natively meant SCALE. Running Storj on the R630 would have meant a manual container deployment, or a Docker VM alongside CORE, or something else that would carry more operational burden than clicking Install on the SCALE app catalog.

So I added SCALE. Not because I preferred it as a platform. Because the workload chose the platform.

SCALE went onto the Dell 740xd — a newer chassis than the R630, running SCALE as a VM on Proxmox on that machine. That let SCALE coexist with other workloads on the same physical server without turning it into a dedicated NAS. Storj lives there. MMX has some workload there too, alongside CORE's harvester. And over time, personal storage — media files, personal NAS shares — landed on SCALE because that's where the newer capacity was, and because SCALE's app catalog was better developed for the shape of workloads that came later.

CORE was chosen for XCH. SCALE was added for Storj. Everything else came in based on whichever server had space and shape at the time it needed to land.

// the split as it stands

Here's the current arrangement in the shortest form.

CORE (R630): XCH farming, MMX farming. Storage is JBOD-backed, structured as multiple small pools shaped by the historical HBA ceiling. The server has more capacity than the workloads currently need.

SCALE (740xd, as a Proxmox VM): Storj, some MMX, personal NAS, media files. Multiple pools, mixed drive shapes, one larger pool separating personal storage from DePIN workload storage.

No replication between the two servers. No cross-platform snapshot strategy today. Backups on both are on the horizon as part of a separate Proxmox Backup Server project — the natural place for that to land is one PBS instance handling both TrueNAS deployments plus everything else in the rack, rather than each TrueNAS doing its own thing.

The two servers don't talk to each other beyond being on the same network. They're not a cluster, they're not a pair. They're two independent deployments that happen to share a rack.

// what's shaped this beyond design

Four things shaped the current arrangement that weren't design choices.

The HBA drive-count ceiling. Real constraint in the early years. Made "many small pools" the shape rather than "one large pool with stacked vdevs." That shape persisted after the constraint stopped mattering — because reshaping ZFS pools is destructive, and there was never a moment worth spending the migration cost.

The XCH-to-alternatives pivot. When XCH scaled down as a workload, the drives went with it. What was left was capacity, not deliberate architecture. If XCH had stayed the primary workload, I probably wouldn't have added SCALE at all — the R630 alone was doing what needed to be done.

Storj's app requiring SCALE. This is the specific technical trigger. Storj's TrueNAS integration was on SCALE. If Storj had shipped a CORE plugin, or if I'd been willing to run Storj in a manually-managed container on CORE, SCALE might never have come in. One workload's platform choice became the reason for a second TrueNAS deployment.

Accretion of what came later. Personal storage, media files, the second-order workloads — they landed on SCALE not because SCALE was the right platform for them, but because SCALE was newer and had capacity. That's not architecture. That's opportunistic filling of empty space.

The pattern that connects all four: each specific arrival made sense on its own; the aggregate is a shape nobody would have designed from a blank sheet.

That's not a failure. It's what actually happens when you run infrastructure for years without a clean-slate migration event.

// the consolidation question

The natural next question: would I consolidate?

The criterion I'd apply is hardware footprint. If consolidation reduced the number of physical servers running TrueNAS in the rack, it would be worth doing. If it just moved workloads around without changing the physical footprint, it wouldn't.

That criterion doesn't currently trigger. Here's why.

The R630 is dedicated to CORE. Consolidating CORE's workloads onto SCALE would free the R630 — real hardware footprint reduction. But it would require migrating XCH and MMX plot data to SCALE, which is terabytes of data and a maintenance window I don't have. Both are non-trivial.

The 740xd runs SCALE as a VM alongside other workloads. Consolidating SCALE's workloads back onto CORE would free the SCALE VM but not any physical hardware — the 740xd would still be running other things. So the consolidation gain there is smaller.

Both servers currently run stable. No degradation pressure, no failing hardware, no capacity ceiling that forces the question. The forcing function isn't present.

PBS project is the natural precursor. Before consolidating anything, I want backup and snapshot posture solid across both platforms. That's the Proxmox Backup Server project sitting alongside this one on the roadmap. Consolidating without settled backup posture would be a much bigger risk than consolidating with it in place.

So the status is: not planned, not off the table. If the R630 develops a fault that would require investment to fix, that's the moment consolidation becomes real. Until then, both run.

The under-utilised capacity is what makes consolidation worth thinking about. The migration blast radius and the missing backup posture are what make it worth deferring.

Both those things are true simultaneously, and that's fine. Operator-honest posture doesn't require every decision to be resolved. Some decisions are open by design.

// closing

Two TrueNAS servers in one rack because history shaped it that way. CORE for XCH first, SCALE added because Storj required it, workloads accreted onto both from there.

The coming quarter doesn't change either of those facts. What changes is the PBS project settling backup and snapshot posture on both platforms — the piece that has to happen before consolidation is even a coherent question. After that, a hardware event, a workload consolidation, or a maintenance window I don't have yet.

Until then, both run.

If you're running a homelab TrueNAS today and wondering whether to migrate from CORE to SCALE — or the reverse — the answer isn't in a platform comparison. It's in your workload stack. Whether the apps you run natively on one but not the other justify the migration cost, and whether your backup posture is ready to make the migration safe. If either answer is no, the honest posture is to leave it where it is.

Sometimes the operator-grade decision is to leave things the way you found them and put the migration budget elsewhere. This is one of those decisions. Both TrueNAS servers run, and probably will for some time.

OPNsense on bare metal — 18 months as the only perimeter

JJozzieTech — Thu, 16 Jul 2026 20:32:03 +0000

For most of the homelab's life, there wasn't a firewall.

The ISP router did what ISP routers do — NAT out, block unsolicited inbound, don't ask hard questions. That's not zero security. It's just not intent-driven security. Which was fine, because the rack ran things I wrote for myself, and nobody on the internet was paying attention to my home connection.

Then I stood up an Akash provider.

The segmentation piece walked through what changed inside the rack when Akash landed — VLANs, ACLs, east-west policy. This piece is the parallel story of what changed at the edge. OPNsense on a dedicated box, ISP terminating on the WAN interface, all internal traffic on a single trunked LAN interface, running for 18 months as the only thing between the rack and the internet.

I'll cover the origin, the hardware, the split of labour between OPNsense and the Nexus 3548P, the four war stories that taught me the platform, and the four things I still haven't done that I know I should.

The last section is the point of the piece. Retrospectives that only list what worked are marketing. Retrospectives that admit what's still on the list are useful.

// no firewall, then one firewall

Before Akash, the perimeter was the ISP router.

Not zero security. Consumer-grade routers do block unsolicited inbound by default — that's been true of every ISP-supplied box shipped in the last twenty years. But the operative word is default. There was no policy. The perimeter existed because the ISP router existed. Anything more specific — logging, per-service inbound rules, outbound restriction, awareness of what was actually crossing — didn't exist because I hadn't asked for it.

That worked for a rack running things I wrote for me, running when I was watching. When Akash landed, the calculus changed. The Akash provider accepts inbound tenant deployments at an ingress that is by design publicly discoverable on-chain. Trusting the ISP router to be the only thing between that ingress and my workstation stopped being defensible once the provider was live and taking inbound deployments.

The natural sequence, once I was building for that: segment first, then police. The segmentation piece covered the segment step. This is the police step.

OPNsense went in on a dedicated box. ISP terminated on its WAN interface. Every internal VLAN — management, workload VLANs, storage VLAN — trunked into a single LAN interface. The rack now had two policy-driven layers separating it from the internet: the segmentation inside, and OPNsense at the edge. Neither trusts the other by default.

// why bare metal, not a VM

The obvious cheap answer would have been an OPNsense VM on Proxmox. It would have run fine. I didn't do that. Three reasons, in the order I actually thought about them.

Compute belongs to workloads. The Proxmox hosts exist to run DePIN infrastructure — Akash, storage, harvesters, validators. Every VM slot I give to a perimeter firewall is a slot I don't give to something earning. Bare metal at the edge means the servers stay full of what they're for.

The perimeter is a different animal from the compute. The firewall shouldn't share a fate with the hypervisor. If Proxmox goes sideways — kernel panic, storage-driver bug, upgrade gone wrong — the perimeter should be uninvolved. Bare metal enforces that separation physically, not just conceptually.

ISP terminates here. OPNsense is the box the ISP handoff talks to. That relationship is cleaner when exactly one physical device owns it, and when that device isn't a guest on a hypervisor whose primary job is something else entirely.

The hardware, as it exists in the rack today:

model:      dell optiplex 7080 SFF
cpu:        intel core i7-10700 (8c/16t)
memory:     32 gb
storage:    SATA SSD, built-in bay
nics:       1× onboard intel i219  (WAN)
            4× intel i350 add-in card  (LAN + 3 spares)
os:         opnsense 25.7 (freebsd base)
form:       SFF, rack shelf

The i7-10700 is overkill for what OPNsense currently asks of it. This rack doesn't push enough traffic to stress an 8-core Comet Lake at the edge. But headroom is what enables the still-on-the-list items — IDS at line rate needs cores, and a CARP secondary needs a matched-pair budget to be defensible.

Between OPNsense and pfSense: open-source posture and the UI. The plugin ecosystem sealed it.

// the split of labour

The segmentation piece established the split. Sharpening it for this piece:

The Nexus 3548P handles east-west. Inter-VLAN routing at line rate on the switching ASIC. Workload-to-workload isolation lives on the Nexus, enforced by ACLs on each SVI.

OPNsense handles north-south. WAN, NAT, port-forwards, outbound NAT posture, inbound perimeter policy. Anything that crosses the boundary between the rack and the internet.

Two rulesets, two devices, two audit surfaces. Both stay simpler.

Interface layout at the OPNsense end:

WAN interface   →  ISP handoff, single link, dhcp from provider
LAN interface   →  tagged trunk to nexus, carries all internal vlans
spare ports     →  available for CARP secondary or dedicated management

One physical LAN interface is enough because the workload VLANs are tagged on the trunk. OPNsense sees each VLAN as a virtual interface on top of the physical LAN port and can write rules against each independently — inbound and outbound, per-VLAN, without the physical port count constraining what's possible.

The spare ports on the add-in card matter for what's coming. When the CARP secondary goes in, pfsync between the pair needs its own link. That's what one of the spares becomes.

// what the rack does with NAT

OPNsense defaults to Automatic outbound NAT: one masquerade rule per LAN interface, everything else inferred. That's what most homelab OPNsense builds run. It's the correct default. It also isn't what this rack runs.

This rack runs Manual outbound NAT. Every workload VLAN has an explicit outbound rule referencing a named network alias. The pattern:

alias:  workload-a-net   →   10.x.x.0/24

rule:   source workload-a-net   →   NAT to WAN interface address

Two reasons that's worth the extra config surface.

Explicit is better than inferred. When you're staring at a firewall trying to work out whether workload X can reach the internet, seeing an explicit named rule beats reasoning about Automatic-mode inference every time. The rule either exists or it doesn't. There's no "well, in Automatic mode it would generate…" middle ground.

Aliases are reusable. The same workload-a-net alias that appears in the outbound NAT rule can be referenced in inbound rules, port-forwards, floating rules — anywhere the subnet needs to appear. One object, one place to update. When a workload's subnet changes — or when it moves — one alias edit propagates everywhere.

The trade-off: Manual mode means new workloads don't get NAT for free. Adding a workload means adding an alias, adding a rule, and applying. That friction is a feature. It means every workload on the wire had a moment where someone explicitly decided how it talks to the internet. Nothing gets to the outside by accident.

// four things OPNsense taught me

Four incidents from 18 months in. Each taught a different class of lesson.

1. The saved config that wasn't the applied config.

A static route to a workload subnet, correctly configured in OPNsense — and workloads on that subnet couldn't reach the internet. Firewall rules checked out. NAT checked out. The routing table on the L3 switch was correct. The route was in the OPNsense config screen but not in the OPNsense routing table.

Pending changes had been staged and never applied. Clicking Apply activated the route and connectivity came back immediately.

Lesson: saved is not applied. OPNsense stages changes to firewall, NAT, and routing configs, and requires an explicit Apply to activate them. The running config can silently drift from the staged one if you forget. Any troubleshooting session that starts with "but the config looks right" should also check whether pending changes are waiting.

2. The subnet OPNsense wouldn't accept as a NAT source.

Configuring outbound NAT for a workload VLAN. Enter the workload subnet as the source. OPNsense refuses: "not a valid source IP address or alias."

The subnet was valid. The rule editor just wouldn't accept it in raw form.

The fix: create a named network alias containing the subnet, then reference the alias in the rule. Alias workload-a-net containing 10.x.x.0/24, and the rule takes it without complaint.

Lesson: OPNsense treats aliases as first-class objects. Some rule fields insist on aliases even when a raw CIDR is technically valid. This isn't a bug — it's the OPNsense-idiom answer to "how do we make firewall rules maintainable at scale." Once the reflex to reach for aliases is in place, everything gets more consistent.

3. The firewall rule that wouldn't edit.

An inbound WAN rule showed up in the firewall ruleset but the edit UI wouldn't let me change it. It looked like a normal rule. It just wouldn't respond to edits.

The rule was auto-generated by a NAT port-forward. OPNsense's port-forwards have a Filter rule association setting that automatically creates the corresponding firewall rule and manages it as a dependency of the port-forward. Direct edits either don't apply or get overwritten the next time the port-forward is touched.

Fix: either edit the port-forward's own rule fields (which flow through to the generated firewall rule), or set the association to "None" and manage the firewall rule separately.

Lesson: know which rules OPNsense controls and which you control. Not every rule in the ruleset is an independent object. Assuming otherwise costs you an afternoon.

4. The package update that wouldn't complete.

A routine package and firmware update ran into a repository / package-consistency issue and stopped mid-upgrade. Recovering the box required diagnosing the state of the pending upgrade rather than just retrying. Endpoint landed at OPNsense 25.7.3_7 eventually.

The firewall stayed on-line during the diagnosis. That was luck. I hadn't taken a config backup before the update, and I hadn't confirmed console access. If the update had left the box in an unusable state, I would have been rebuilding OPNsense from memory.

Lesson: before touching updates, capture a config backup and confirm console access. Both take five minutes. Neither had happened. Both are habits now.

// what I still haven't done

Four gaps. In the order I'd actually tackle them if I started this weekend.

1. Backups first.

The single most embarrassing gap. OPNsense's config is an XML export that can be committed to Git, uploaded to cloud storage, or scheduled to a network share via the built-in backup module. None of that has happened here.

Backups are first not because they're the most urgent to have, but because they're the cheapest to get. Ten minutes of config setup and I have a restorable state. Every subsequent gap-closing effort is safer once this is in place. The next OPNsense update — and it'll happen — will be less nerve-wracking with a known-good config sitting in a repo somewhere.

2. HA second.

Would I make the same bare-metal call again? Absolutely. Would I have planned CARP into the initial build from day one? Also absolutely.

Retrofitting HA to a running perimeter is harder than designing it in. CARP virtual IPs need to be planned around the WAN handoff. pfsync between peers needs a dedicated link. Both are easier to reason about with a matched pair of boxes than with an active production box and a second box being brought into a pair. The next OptiPlex is the plan. It's a project, not a config change.

3. Logging third.

OPNsense's logs currently live on the box and roll over. That's fine for "check the last hour's blocked traffic." It isn't fine for anything else. The natural target is either a Loki/Grafana stack in the rack or Graylog on a dedicated VM.

Logging goes before IDS/IPS because IDS/IPS generates the logs. Having nowhere for them to land defeats the point.

4. IDS/IPS last.

Suricata on the LAN interface, ET Open ruleset, IDS mode first (observation) before IPS mode (enforcement). Deliberately last, because the false-positive load is real, the tuning effort is real, and the payoff without steps 1–3 is limited. IDS/IPS without a config backup, without HA, and without a place to send logs would just add fragility instead of catching it.

The order is deliberate. The temptation with lists like this is to lead with the exciting-sounding gap. Backups aren't exciting. They're the load-bearing gap that everything else depends on.

// closing

The perimeter has held for 18 months. Zero incidents I've traced to the firewall itself. Two of the four war stories above (the un-applied config, the un-editable rule) taught me how the UI thinks. Two (the alias requirement, the package update) taught me how the platform thinks. All four made me more useful, not less.

The next 18 months aren't going to be about new capability. They're going to be about closing the four gaps. A backup piece. A CARP-HA piece. A logging-to-something-real piece. Eventually an IDS/IPS piece, once the first three are in place to catch what it finds.

If you're running a homelab without a firewall today because the ISP router is fine, you're where I was two years ago. The trigger to move isn't a security incident — those don't announce themselves. The trigger is any workload you'd rather not have sharing a box with your workstation, or any inbound traffic you can't defend by hand-waving at NAT.

If you already have OPNsense running: check whether your config is backed up. That's the single highest-leverage thing you can do this week.

How I got a suspended TrueNAS pool back online without replacing a single disk

JJozzieTech — Thu, 09 Jul 2026 02:54:20 +0000

The rack had just come back online after an outage. Most things looked fine — VMs booted, the network reconverged, monitoring caught up on the missed intervals. Then I ran zpool status on one of the TrueNAS SCALE nodes and saw output I hadn't seen in production before.

An 11-disk RAIDZ1 vdev. Multiple drives marked as faulted. One member listed as removed. The pool itself in SUSPENDED state.

The reflex, in that moment, is to start replacing disks. That reflex is wrong — at least as the first move. This piece is how I got the pool back online without touching a single drive, and what I learned about how ZFS reports failure states along the way.

The pool in question is on my TrueNAS SCALE machine, which runs alongside a separate CORE box on a different chassis. The recovery pattern is the same on either — this isn't SCALE-specific ZFS behaviour — but the incident happened on SCALE, so that's where the console output in this piece comes from. For privacy, I'll refer to the affected pool as MediaPool-A.

// what the console showed

The output was the classic suspended-pool warning:

  state: SUSPENDED
 status: One or more devices are faulted in response to IO failures.
 action: Make sure the affected devices are connected, then run 'zpool clear'.

Then the vdev listing. The RAIDZ1 topology looked much worse than it turned out to be — most of the 11 disks were showing as faulted, and one was marked as removed. If you'd shown me that output out of context and asked me to guess what happened, I'd have said "shelf died" or "backplane failure." The vdev looked shot.

But there was a shape to the failure that mattered. Multiple disks going down at the same time is exactly what you'd expect if something they shared had gone down — an HBA path, a SAS cable, a backplane, a controller — rather than the disks themselves failing individually. Simultaneous multi-disk failure is one of those things that sounds catastrophic and usually isn't. Independent failures don't cluster in time. Shared-path failures do.

That's the diagnostic reframe the piece is built around. Before you touch a disk, ask what those disks have in common that isn't the disks.

// the reflex to resist

The most important part of ZFS recovery is what you don't do in the first ten minutes. Here's the list, in order of danger:

Don't add disks to the pool. Whatever's shown as unassigned in the TrueNAS GUI may not actually be unused. If the pool is suspended and the disks are still recognised by the OS, they're still valid pool members even if the GUI is being coy about it. Adding them to a new vdev structure is the fastest way to permanently lose the pool.
Don't wipe any disk. Same reasoning. If a disk still has valid ZFS labels for this pool, wiping it is destructive.
Don't replace any disk. Not until you've confirmed there's a disk that actually needs replacing. See earlier point on multi-disk-fault patterns.
Don't export or destroy the pool. zpool destroy is exactly what it sounds like. zpool export is safer but shouldn't be done while the pool is in an unstable state.
Don't start a scrub while the pool is suspended. A scrub is not the recovery mechanism when the pool has been suspended due to I/O access problems. The pool needs to be brought back into a stable online state first. Scrubbing before that either does nothing useful or, worse, hammers the disks while they're on an unreliable path.

That last one catches people. Scrubs are the go-to "is everything OK?" tool in ZFS-land, and the muscle memory when something looks wrong is to run one. Muscle memory is wrong here. A scrub is validation, not recovery. It belongs at the end of the recovery process, not the start.

// step one: are the disks actually there?

Before running anything against the pool, I checked whether the operating system could see the disks. This is the cheapest possible diagnostic and it usually tells you which world you're in.

smartctl --scan

The disks were visible. All of them. That was the first good sign — it meant the OS could see the hardware, which meant whatever had happened during the outage had resolved at the hardware/path level, which meant the pool was recoverable in software without touching any physical component.

Then I checked kernel logs for signs of ongoing HBA or SAS trouble:

dmesg -T | grep -iE "sas|scsi|mpt|mps|mpr|reset|timeout|failed|medium error|sense|I/O error|device offline|link"

The kernel log showed events from the outage itself — expected, since the disks had briefly gone away and come back — but nothing recent. No active storms of SAS resets. No timeouts still firing. No disks going offline. The path was stable now, whatever had happened during the outage.

That was the second good sign, and it was the one that gave me confidence to move to the recovery step. If the kernel had still been throwing SAS resets or timeouts in the current second, I would have stopped, escalated to the physical layer — cables, HBA, PSU — and only come back to ZFS after the path was clean. Running zpool clear against a still-unstable path just re-triggers the fault, and does it at higher urgency this time.

// zpool clear, safely

Once the two "is the path stable?" checks came back clean, the recovery command was the one ZFS itself had already suggested:

zpool clear MediaPool-A
zpool status -v MediaPool-A

The pool returned to an ONLINE state. One disk briefly showed a small number of checksum errors on the first status check after the clear — expected, because the outage had generated some genuinely bad reads that ZFS had queued up. A second zpool clear zeroed the counters, and the follow-up status came back clean:

MediaPool-A = ONLINE
all disks = ONLINE
READ / WRITE / CKSUM = 0
errors = No known data errors

No files reported as corrupted. No permanent errors. The pool was back.

The recovery itself was two commands. The rest of the work was in the "is the path actually stable?" checks before them — cheap to run, and the difference between a clean clear and re-triggering the fault at a moment when the pool was already suspended.

// what actually happened

The most likely cause was a temporary I/O path disruption during the outage. Something in the shared path — HBA, cable, expander, backplane, or the disks all briefly losing power at slightly different times as the rack came back — presented as multiple simultaneous device failures to ZFS. ZFS did exactly what it's designed to do: suspended the pool to protect the data from being written to devices it wasn't sure were healthy.

Once the path stabilised, the disks were visible again. zpool clear told ZFS "the fault condition is resolved, retry," and ZFS did. Because the disks and their data were still intact underneath, the retry succeeded, and the pool came back with no permanent state loss.

The three ZFS terms that mattered here are worth stating precisely, because their intuitive meanings are misleading:

FAULTED does not always mean a dead disk. It means ZFS has decided this device is unreliable enough to stop trusting for the moment, based on the errors it's seen. A device can be FAULTED for reasons that have nothing to do with the disk itself.
REMOVED does not always mean physically removed forever. It means the device that was here at some point isn't visible right now. It can be transient.
SUSPENDED does not always mean the pool is lost. It means ZFS has stopped I/O on the pool to protect it, pending resolution of whatever fault condition triggered the suspension.

Context is what tells you which of those interpretations applies. Multiple simultaneous FAULTED devices, immediately after a rack outage, on drives you can still see with smartctl and don't have active kernel errors on, points at "transient path disruption during the outage" and away from "disks died." A single FAULTED device weeks after normal operation points the other way, and the response is completely different.

// what I'd make instinctive

Two things, and one thing I already had right.

Look at the shared layer first. When multiple disks fault at once, the useful question is not "which disks failed?" but "what do they share that could have failed?" HBA, SAS path, backplane, PSU, controller firmware. Independent failures don't cluster in time; shared-path failures do. Every panic-response-to-multi-disk-fault I've ever done has, eventually, come back to something at the shared layer. Skipping straight to the shared layer as the first diagnostic saves you the panic.

zpool clear is safe only after the path is confirmed stable. It is not a magic fix. If the underlying path is still unstable, zpool clear re-triggers the fault immediately, and the second suspension arrives faster than the first because ZFS has now seen the pattern twice. Confirming path stability first — smartctl --scan, then dmesg | grep for active errors — takes about thirty seconds and is what makes zpool clear a recovery tool instead of a way to make things worse.

The thing I had right: not scrubbing while the pool was suspended. That's an instinct worth keeping. Scrub validates a stable pool. It doesn't recover an unstable one.

// closing

This was a clean recovery because the underlying condition turned out to be transient. The pool came back with no touch to any physical disk, no wipe of any data, no rebuild of any vdev. The recovery cost was two commands and about ten minutes.

Not every ZFS recovery is that clean. The same outage that suspended MediaPool-A also took down a second pool on the same box in a different, harder way — one where an entire RAIDZ1 vdev appeared to be gone, and the recovery required reading ZFS labels directly to prove the disks were still there before running any of the commands above. That's the next piece.

If you're recovering from something like this now: check the disks are visible, check the kernel logs are quiet, then run zpool clear. In that order. Everything else you might be tempted to do can wait.

I run my homelab like a miniature data centre — here's the network design that made it possible

JJozzieTech — Thu, 02 Jul 2026 06:57:36 +0000

The homelab started flat. One /24, everything on it. My workstation, the NAS, the Proxmox host, and — over time — a growing list of workloads sharing the same broadcast domain because that was the path of least resistance.

For a while, that was fine. A homelab running one workload doesn't need segmentation any more than a house needs an office door.

Then I stood up an Akash provider.

An Akash provider is, in shape, a Kubernetes cluster that accepts inbound tenant workloads from the internet — real deployments, paying for compute, containers I didn't write landing in namespaces on my hardware. The provider itself is documented at github.com/jjozzietech/akash-provider-ops-public — this piece is about the network underneath it. The containerisation posture itself is fine. I trust the isolation model. But trust isn't a network design. And the network at that moment had the tenant workload cluster sitting on the same subnet as my workstation, my NAS, and my Proxmox management interface.

That was the moment I stopped thinking of the rack as a home network with extra boxes, and started thinking of it as a small data centre.

This piece is the network design that came out of that shift. I'll cover the layout, the rules that hold it together, and the Nexus and Proxmox configs that anchor it — with the specifics of my own deployment sanitised. It's not a step-by-step replication guide. It's the design pattern, with enough of the shape to be useful and enough restraint to not double as a recon document for my own rack.

// the original design

The flat layout looked like this:
home lan — 192.168.1.0/24

opnsense (perimeter)

cisco nexus (dumb L2 switching)

proxmox host

workload VMs (all on the same subnet)

What it got right: zero routing complexity, everything reachable from everywhere, fast to stand up. If you're running one project on a homelab, this is the correct design. Don't over-engineer it.

What stopped working, as soon as the second project landed on the rack, was that the design had no answer to three questions:

How do I write different firewall policies for different workloads when they all share a subnet?
How do I keep a compromise in workload A from reaching workload B?
How do I troubleshoot when every packet in the rack lives in the same broadcast domain?

The last one is the operational tell. When "which workload is chatty on the wire right now?" becomes a question you can't answer from a switchport counter, the design has outgrown the problem it was designed for.

The industry term for what I was worried about is east-west traffic — traffic between workloads on the same tier, as opposed to north-south traffic between workloads and the outside world. Flat L2 doesn't give you any policy handle on east-west. You just have to trust that nothing on the wire behaves badly.

That's a defensible assumption for a house LAN. It's not a defensible assumption for infrastructure that hosts tenant workloads.

// the reframe: stop thinking residential

Commercial data centres don't put every workload in the same subnet. They isolate services into dedicated network segments, each with its own routing policy, its own firewall posture, its own blast radius. When something goes wrong, the fault is bounded. When something needs a different security posture, it gets one. When a workload needs to move to different hardware or a different site, the network context comes with it.

That's the model I took. Every project on the rack gets:

Its own VLAN
Its own /24
Its own firewall policy at the routing boundary
No default trust with any other VLAN

The hypervisor stays on management. Workloads live on their own VLANs. Ops access is explicit, not implicit.

A quick note on hardware, because it matters. The switch doing this work is a Cisco Nexus 3548P — enterprise-grade L3 switching, ACLs applied inbound on the SVI, inter-VLAN routing at line rate on the switching ASIC rather than punted to a CPU. A UniFi or Mikrotik can do most of what's below, and if you're building this pattern on consumer gear, that's a reasonable place to start. What you give up is the ability to enforce policy at wire speed with the ACL evaluated in hardware. On a homelab-sized rack it's rarely the bottleneck. On a rack that hosts real workloads with real tenants, I wanted the ceiling raised.

// the architecture

Here's the layout, in the same YAML metadata style I use across the site:
management: 192.168.1.0/24 workstation, opnsense, proxmox host

vlan N (workload-a): 10.N.N.0/24 first project

vlan M (workload-b): 10.M.M.0/24 second project

vlan …: … future

Actual VLAN IDs and subnets in my deployment differ. The pattern is what matters, and I'll get to why the pattern is the way it is in the "what I'd do differently" section.

Three design rules underpin this.

The hypervisor stays on management, VMs live on workload VLANs. The Proxmox host is not a workload. It's the platform workloads run on. Putting it on the management VLAN means the host stays reachable when a workload VLAN goes sideways — backups continue, console access works, snapshots run. If the Proxmox host is on the same VLAN as a workload and that VLAN's routing breaks, you've just lost your recovery path.

One /24 per workload, gateway on the Nexus. The L3 boundary between VLANs is where policy lives. That means the Nexus SVI, not OPNsense. OPNsense's job is different, and I'll get to it.

Workload VLANs are peers, and no VLAN trusts another by default. East-west is denied. ACLs grant the exceptions. If workload A ever needs to talk to workload B, that's a deliberate rule, not a side effect of shared broadcast domain.

// what the nexus does

Three sanitised snippets, each with a sentence on intent.

VLAN definition:
vlan 10

name workload-a

Nothing exciting. The VLAN is a container; the interesting policy lives on the SVI.

SVI with gateway and inbound ACL:
interface Vlan10

description workload-a-gw

ip address 10.10.10.1/24

ip access-group acl-workload-a in

no shutdown

The ip access-group acl-workload-a in line is the important one. Inbound on the SVI means traffic entering the Nexus from the workload VLAN is what gets filtered. Any packet leaving workload A's subnet passes through that ACL before it gets routed anywhere else. That's how east-west enforcement happens at the routing boundary.

Trunk to a Proxmox node:
interface Ethernet1/3

description pve01-trunk

switchport mode trunk

switchport trunk allowed vlan 10,20,30

spanning-tree port type edge trunk

no shutdown

The switchport trunk allowed vlan list is explicit. Only VLANs I've deliberately added to the trunk can traverse it. New workload comes online → the VLAN gets added to the allow-list. Mundane, but it means the trunk isn't a wide-open pipe by default.

I leave the native VLAN unset on workload trunks. Tagging is explicit on the Proxmox side, which I'll cover shortly.

// ACLs are the actual security mechanism

VLAN segmentation isolates broadcast domains. That's it. Two VLANs on the same switch, both trunked to the same Proxmox host, can absolutely talk to each other — as soon as one of them has a route to the other. The route is provided by the SVI. Which is why the SVI is where policy has to live.

The pattern, not the rules, for a workload VLAN's ACL looks like this:
acl-workload-a:

permit management-workstation -> workload-a (ops access in)

permit workload-a -> dns, ntp (outbound essentials)

deny workload-a -> 10.0.0.0/8 (block east-west)

deny workload-a -> 172.16.0.0/12 (block east-west)

deny workload-a -> 192.168.0.0/16 (block east-west)

permit workload-a -> any (internet last)

Walking through the logic:

Rule order matters. The three RFC1918 denies sit before the permit-any-internet. Reverse them and the denies never evaluate — the permit-any matches first and every internal network is reachable. This is the single most common way I've seen this pattern implemented incorrectly.

The management workstation gets an explicit permit at the top. Without it, I lose ops access to the workload VLAN — SSH, monitoring, any control-plane traffic. With it, only that one host can reach in. Everything else that tries to reach workload A from a private range hits the RFC1918 denies further down.

DNS and NTP get their own permit lines. Cleaner than carving exceptions into the deny block. The workload needs to resolve names and know what time it is; both are explicit.

No VLAN-to-VLAN permits exist. If workload A needs to talk to workload B in the future, that'll be a deliberate exception — added, logged, reviewed. Default deny is the actual default, not aspirational.

What's not in this article: the per-host permits, the port restrictions inside the workload VLAN, the exact management-range definition, and the north-south inbound permits handled at the OPNsense boundary. The shape is the point. The full ruleset stays on the rack.

// dividing the labour: nexus vs opnsense

There's a clean split between the two devices.

The Nexus handles east-west: inter-VLAN routing and policy. Everything that stays inside the rack.

OPNsense handles north-south: WAN, NAT, port forwards, outbound NAT, anything that crosses the perimeter to or from the internet.

Two devices, two audit surfaces, two separate rulesets. Both stay simpler as a result.

The alternative — collapsing everything into OPNsense as a router-on-a-stick — is a design I see in a lot of homelab writeups, and it works. But it has two costs. First, inter-VLAN traffic becomes CPU-bound on the firewall instead of running at line rate on the switch. Second, north-south and east-west rules end up in the same ruleset, which makes both harder to reason about over time. You end up asking "does this rule apply to WAN traffic or inter-VLAN?" every time you touch the config, and the answer depends on interface assignments buried elsewhere.

Keeping the two roles physically separate — Nexus for east-west, OPNsense for north-south — is more disciplined. And on the Nexus, inter-VLAN routing doesn't touch the firewall at all, which means an OPNsense reboot doesn't interrupt workload-to-workload traffic. (It also doesn't enable workload-to-workload traffic, because the Nexus ACL denies it by default. But the point is that the firewall isn't in the east-west path either way.)

// the proxmox side

The Proxmox network config on the host is deliberately simple. One VLAN-aware Linux bridge, one LACP bond feeding it, management IP on the untagged side.

Here's the shape, sanitised — real interface names, redacted IPs:
auto lo

iface lo inet loopback
iface ens7f0 inet manual

iface ens7f1 inet manual
auto bond0

iface bond0 inet manual

bond-slaves ens7f0 ens7f1

bond-miimon 100

bond-mode 802.3ad

bond-xmit-hash-policy layer2+3
auto vmbr0

iface vmbr0 inet static

address 192.168.1.10/24

gateway 192.168.1.1

bridge-ports bond0

bridge-stp off

bridge-fd 0

bridge-vlan-aware yes

bridge-vids 2-4094

Two physical NICs (ens7f0 and ens7f1) bond together into bond0 using LACP (802.3ad) — the switch side is configured as a matching port-channel. That gives me two paths to the Nexus, active/active, with automatic failover if one link drops.

The bond feeds a single Linux bridge, vmbr0, configured as VLAN-aware. The bridge trunks every VLAN from 2 to 4094 (in practice, only the VLANs actually on the switch's allow-list traverse it). The hypervisor's management IP sits on the bridge itself on the untagged VLAN — which is the management VLAN, reachable from the workstation and the rest of the management infrastructure.

The VM side is one line of config per VM: the NIC gets a VLAN tag set in the Proxmox GUI. The VM sees a regular Ethernet interface. The bridge handles tagging on the way out to the bond, and the Nexus routes based on the tag. That's the whole story on the host.

Notably absent: a separate bridge per VLAN. That design works, but it doesn't scale — every new VLAN means a new bridge, a new set of uplinks to manage, and a growing surface of things that can drift out of sync between hosts. One VLAN-aware bridge trunked to the switch does the same job with dramatically less state to maintain.

// what I'd do differently

Two things: one process, one design. Plus the honest note on the migration itself.

Process: design the ACLs before the workload goes live, not after.

The segmentation itself I got right the first time. The ACLs I got right eventually — but I built them reactively, as workloads landed and I discovered gaps. New workload goes live → I stand up the VLAN and SVI → I permit the ops access I need to configure it → the workload comes up → I notice traffic leaking somewhere it shouldn't → I tighten the ACL. Each iteration got closer to correct. But each iteration also had a window where the ACL was more permissive than I wanted, and one of those windows was long enough to matter.

The right approach is to have a template for both directions of policy before you provision anything. The east-west deny block as a ready-to-paste chunk. The north-south permit list — which ports OPNsense will forward inbound, from where — drafted alongside the workload's own deployment plan. Apply both at the moment the VLAN is created. The workload never exists in a permissive state, even briefly. The "did I remember to tighten that rule?" question never comes up.

This is the single thing that separates the segmentation I have now from segmentation I'd have designed on day one if I could start over.

Design: I'd keep the VLAN scheme exactly as it is.

The pattern is VLAN N maps to 10.N.N.0/24. VLAN 10 lives in 10.10.10.0/24. VLAN 11 lives in 10.11.11.0/24. And so on.

It's opinionated and I'll defend it. The payoff is that troubleshooting is self-documenting. A packet capture shows a host in 10.10.10.0/24 and I know without looking that it's on VLAN 10. An ACL entry references 10.11.11.0/24 and I know without looking it's targeting VLAN 11. Every debugging session, every ACL review, every firewall rule audit is faster because the VLAN ID and the subnet are the same number.

The downside is you're locked into a /24 per VLAN, and you can't ladder easily beyond VLAN ~250 before you start running out of clean second-octet space. For a homelab, that's not a real constraint. If it ever becomes one, the failure mode is switching to a different scheme for the overflow VLANs — the existing ones don't need to change.

Migration: rolling, and it was fine — mostly.

The migration itself was rolling rather than greenfield. I had running workloads, no maintenance window, and no ability to stand up a parallel network to cut over to. So each project moved one at a time: stand up the new VLAN, ACL, and SVI on the Nexus; add the VLAN to the Proxmox trunk allow-list; retag the VM's NIC; update any OPNsense port forwards to the new IP; verify; move on to the next.

Here's the one that stung. My Storj storage node went dark for longer than I'd like to admit during the TrueNAS cutover. TrueNAS moved from the flat network to its own VLAN — new subnet, new IP. The Storj app runs as a TrueNAS CORE plugin, so it inherited the new IP automatically. What didn't inherit anything automatically was the OPNsense port-forward, which cheerfully continued sending inbound Storj traffic to TrueNAS's old address. From my side, everything looked fine — TrueNAS was up, the plugin was running, SMB shares were serving internal clients without complaint. From the Storj network's side, my node had stopped accepting inbound connections. It took a reputation dip before I noticed.

The failure mode is generic and worth naming: when a workload moves subnets, every downstream reference to its old address is now stale. Port-forwards, reverse proxies, DNS records, anything with a hardcoded IP in a config, anything scraping Prometheus on the old address. My cutover checklist was fine for the workload itself. It was not fine for everything that talked to the workload. That's the kind of gap a pre-provisioned north-south permit template — with every port and destination mapped out before the move — closes automatically. My checklist now includes an explicit "audit every reference to the old IP" step. It hasn't happened again.

// scaling: why this pays back

The marginal cost of adding workload N+1 is now:

Pick the next VLAN ID and its matching /24.
Define the VLAN and SVI on the Nexus, with the template ACL applied.
Add the VLAN to the Proxmox trunk allow-list.
Tag the VM's NIC.
Configure north-south forwarding on OPNsense if the workload takes inbound traffic.

That's a fifteen-minute job for a workload that doesn't need custom firewall policy, and maybe an hour if it does. Not a redesign. Not a rework of anything already running.

The same logical layout transfers to colocation or a commercial data centre with the physical topology changing but nothing else. The VLAN IDs, the subnets, the ACLs, and the OPNsense rules all move as-is. That was one of the original design constraints — build a homelab network that doesn't need to be rethought when the rack does — and it's the part I'm most quietly proud of.

// closing

Most homelabs don't need any of this. If you're running a Plex box and a media NAS and maybe one game server, flat L2 is the right answer. The complexity here only starts paying back when you're running multiple workloads with materially different trust postures — something you built yourself next to something you didn't, something that accepts inbound traffic from the internet next to your workstation, something in production next to something in perpetual dev.

The moment you cross that line, segmentation stops being over-engineering and starts being basic operational hygiene.

This design pattern is the substrate the rest of the rack runs on. The next few pieces in this series build on it — OPNsense as the perimeter, TrueNAS as the storage layer, and eventually the workload-specific pieces where individual services get their own writeups. All of it assumes the network underneath looks like this.

If you're running mixed workloads on a flat network today, the question isn't whether to segment. It's how many workloads you want to add before you do.