DEV Community: JJozzieTech

The wildcard cert saga — eight days from expiry, three failed attempts, one flag

JJozzieTech — Thu, 27 Aug 2026 19:55:19 +0000

Cert-expiry monitoring is one of those systems that runs quietly for months and then, suddenly, starts asking for your attention with growing volume. On the Akash provider running out of the Sydney rack, the daily-check script had been raising the same warning for a week — quiet drumbeat, background noise until it wasn't. The wildcard cert covering the provider's public ingress had eight days to live.

Eight days is not a crisis. It's not comfortable either. The comfortable window for renewing a wildcard cert that fronts real tenant workloads is measured in weeks, not days, and I did not want to be doing this work at expiry-minus-hours with tenants noticing. So it moved to the top of the queue on 29 July, expiry set at 6 August.

The first instinct was the wrong one. cert-manager is normally auto-magical. It watches its Certificate resources, notices when they're within the renewal window, kicks off a fresh order against Let's Encrypt, and the operator finds out about it via absence — no notification because nothing broke. When the automation goes quiet, the reflex is to assume it just needs a nudge: restart the pod, re-issue the CertificateRequest, watch the reconciler pick things up again.

That reflex was wrong for reasons the first hour of investigation would surface. But the deeper story is what surfaced after that. Three failed renewal attempts. Root causes distributed across four different components — DNS provider, cert-manager, Cloudflare, acme.sh. Rate-limits triggered by the diagnosis itself. And ultimately a fix that landed as a single flag on an already-working tool, once the problem was framed correctly.

This piece is that story. The 8-day countdown, the three attempts that didn't work, the reframing that ended the incident in about half an hour of active work after days of distributed-bug hunting, and the four lessons that stuck.

// the first hour: what killed the renewal cron

Before any renewal attempt, the first hour went to config review. If cert-manager was supposed to auto-renew and hadn't, the question wasn't "what's wrong with the renewal now" — it was "what's been wrong for weeks that stopped the renewal from happening at all."

The acme.sh account configuration held the answer. Credentials were CF_Key + CF_Email — Cloudflare's Global API Key model. Those credentials had degraded, silently rejected for reasons not fully diagnosed at the time. The renewal cron had been executing on schedule the entire time. It had been failing at authentication the entire time. Neither of us — cron nor operator — had noticed. The immediate fix was rebuilding on scoped API Tokens rather than forensically reconstructing the failure cause — an auth-mode swap that doubled as overdue credential rotation.

Discovery mechanism matters. This wasn't found via log review. acme.sh's cron logs did contain terse authentication errors, but they were the kind of error that reads as normal noise if you're not already suspicious. The smoking gun was the config file itself — one line, CF_Key=..., unambiguous once you know Global Keys are being deprecated.

Auth was the technical trigger. The renewal cron dying was the symptom. The structural problem was that nobody was watching whether the cron actually succeeded. The cron was scheduled. The cron ran. The exit code was ignored. The distinction between "the cron executed" and "the cron did what it was supposed to do" was the gap monitoring should have caught, and didn't.

This is the class of failure that turns a routine credential rotation — normally a fifteen-minute job you do during a Cloudflare account maintenance session — into an 8-day fire drill. The credential rotation itself was trivial. The consequence of the credential rotation having happened silently, weeks earlier, without any monitoring signal that it had happened, is what pushed the deadline.

Cron exit codes without monitoring aren't automation. They're theatre.

// combining fixes: credential rotation plus architecture change

The natural response would have been to rotate to a scoped API Token, restart the renewal cron, and consider the incident closed. That's not what happened. The credential rotation got combined with a larger architecture change — one that had been on the "should probably do this eventually" list for months.

The reasoning was structural. Cloudflare's push away from Global API Keys wasn't arbitrary. Global Keys granted full-account write access with no audit trail — one credential capable of every action across every zone on the account, with no way to scope permissions or track which automation had done what. Scoped API Tokens fixed both problems: zone-limited, purpose-specific, individually revocable.

That same reasoning also applied to the DNS-01 challenge path itself. The existing wildcard cert renewal used acme.sh's default TXT placement — write records to the domain's authoritative zone at Cloudflare, remove them after validation. That mode required Global-API-Key-equivalent write access to the entire zone. A safer architecture would isolate the challenge writes to a dedicated CNAME-delegated location — SiteGround holding the primary nameservers, _acme-challenge CNAMEs delegating just the ACME work to Cloudflare, and API Tokens scoped to only that delegated zone.

Doing credential rotation and DNS architecture change as one combined fix meant less integration risk, less cleanup, and fewer moving parts to reason about afterwards. Two-in-one is usually a false economy — but here, the two changes were driven by the same underlying reasoning, and the fix stack for both landed in the same session.

// first failed attempt: cert-manager's zone-walking bug

With API Tokens in place and the CNAME delegation planned, the first renewal attempt went through cert-manager. And this wasn't a single-vector change. Three vectors were shifting at the same time.

The tool was changing: cert-manager as the new standard for the Kubernetes stack, replacing acme.sh. The CA was changing: Let's Encrypt as the new issuer, replacing the incumbent ZeroSSL. The challenge architecture was changing: CNAME delegation replacing acme.sh's default TXT placement.

Each had its own accumulated "should probably do this eventually" reasoning, and two of them were entangled. Wildcard issuance needs DNS-01 regardless of CA — the ZeroSSL setup was already doing that. What made CNAME delegation a prerequisite rather than a nicety was the credential model: default TXT placement required write access to the whole zone, so scoping the token to anything narrower meant moving the challenge records somewhere that existed only for that purpose.

Whether that was a wise bundling decision is worth naming: it was probably too many changes at once. Multi-vector migrations amplify the diagnostic difficulty when they fail, because you don't immediately know which vector is the problem. That was the lesson not yet learned going into the first attempt.

The attempt failed. But not in a way that produced an obvious error.

The failure signature was subtle. cert-manager wasn't returning an obvious error. The CertificateRequest sat in a Pending state — no failure event, no red status, nothing that would light up a dashboard. cert-manager's Cloudflare provider logs showed it querying zones, matching them, and then attempting to place the TXT record for the ACME challenge at a location. And Let's Encrypt's validator was looking for the TXT record at a different location. The challenge never validated. But nothing about the failure looked like a failure — every component was doing what its logs said it was doing.

It wasn't "this is broken." It was "this looks like it's working but the challenge never validates."

That's the class of failure that eats hours. When every log line reads as healthy activity, the diagnostic instinct — "find the error message and follow it" — has nowhere to go. There is no error message.

The trace came from reading cert-manager's Cloudflare provider source directly, understanding its zone-detection logic, and then cross-referencing that logic against the actual DNS zone contents of the affected domain. cert-manager's provider walks the DNS hierarchy from the record name up toward the root, looking for the "authoritative zone" — the zone where it should write the TXT record. That walk followed a CNAME chain. And the CNAME chain led to .snapshots.

.snapshots is a SiteGround backup-access hostname. It's part of the shared-hosting furniture — installed by the hosting provider for their own backup tooling, invisible to you because you didn't set it up and never think about it. If you're not a SiteGround-hosted domain operator you'd never look at it. If you are, you'd still never look at it, because it has nothing to do with anything you actively manage.

Except now it did. cert-manager's zone-walker followed the .snapshots CNAME, decided that was the authoritative zone for the ACME challenge, and placed the TXT record somewhere Let's Encrypt would never look. Every piece was behaving as documented. The whole still didn't work.

The teaching moment isn't about .snapshots specifically. It's about the class of failure where every component reads healthy and the composition fails silently. That class of failure needs a different diagnostic instinct than error-reading. It needs a diagnostic instinct closer to what network engineers use for packet capture — read the actual traffic between components, don't trust either component's log about what it thinks it's doing.

// second failed attempt: the retry loop became the story

With the .snapshots CNAME identified as the zone-walking trap, the second attempt tried a workaround: manually specify the zone cert-manager should target, bypassing the auto-detection. The attempt failed in a very different way.

cert-manager's default behaviour when a challenge fails is to back off and retry. Backoff intervals lengthen with each retry, but the retries never stop until the CertificateRequest is either manually paused or succeeds. That default is correct for transient failures — network hiccups, brief provider outages, momentary rate-limits. It's harmful for persistent failures on already-loaded infrastructure, where the retry adds pressure without being the sole cause of the problem.

The second attempt ran for about twenty minutes before I noticed it had shifted into a distinct failure mode. cert-manager's default backoff made each retry look reasonable in isolation — a single request every few minutes, exponentially spaced. Then Cloudflare 429 responses started appearing in the cert-manager pod logs — rate-limit rejections.

Here the mechanism deserves care. Cloudflare's global rate limit is 1,200 requests per five-minute window, counted account-wide across dashboard sessions, API tokens, and legacy keys combined. When the limit breaks, everything talking to that account gets blocked for the remainder of the window. cert-manager's backoff-driven retries alone weren't generating anything near 1,200 requests in five minutes — but the retry loop was probably enough additional load, on an account already carrying baseline API traffic from other automation, to push the account over the threshold.

Retries were the last straw, not the sole cause. Exact contribution unknown — request-rate telemetry from the incident window wasn't captured, and I can't reconstruct precisely how much of the budget the retry loop was consuming versus what was already in flight.

That makes the lesson more useful, not less. Account-wide rate-limit budgets mean isolated request rates aren't the whole picture. A retry loop that would be harmless on a quiet account can be the tipping point on a busy one. The operator awareness that matters: not "how many requests does this thing generate" but "how many requests does this thing generate on top of what's already there."

The 429s in the cert-manager logs meant other operations against the same account would have been affected during that window too — that's how account-wide limits work by definition. Whether specific other automation on the account actually noticed depends on whether it was making requests during the limited window; I didn't directly correlate specific other-tool failures during the incident.

The retry loop had become part of the problem.

Not the primary cert renewal being wrong — that was still the thing being diagnosed. But the diagnosis, in the form of cert-manager retrying against a persistently failing challenge, was contributing to a rate-limit condition that affected the whole account. Diagnostic activity was making things worse in ways I couldn't fully see.

The immediate fix was to pause the CertificateRequest — one kubectl annotate to stop cert-manager retrying — and wait for the rate-limit window to clear before touching Cloudflare again. That's the mechanical fix. The operator lesson is broader.

Retry loops without circuit breakers can amplify account-wide rate-limit conditions. cert-manager's default retry behaviour is fine for the failure modes it was designed for. It is not fine for the failure mode where the retry adds pressure to a rate-limit budget that's already tight for reasons outside the tool's visibility.

Circuit breakers — "stop retrying after N consecutive failures of the same type" — exist as a concept in most well-designed retry systems. cert-manager's Cloudflare provider doesn't have one aggressive enough to matter in this situation. Knowing that in advance would have saved twenty minutes of adding pressure to an already-loaded account.

// the pivot that wasn't really a pivot

At this point the shape of the incident changed. cert-manager's Cloudflare provider bug — the zone-walking through .snapshots — wasn't going to be a fix I could apply in-session. It was a bug in the provider code, and correcting it would need an upstream patch, testing, release. That timeline was days minimum, and the cert had single-digit days until expiry.

The path forward was cert-manager off the critical path. Which meant collapsing the three-vector migration down to two.

Framing that as a "pivot" isn't quite right. Two of the three vectors — Let's Encrypt as CA, CNAME delegation as challenge architecture — were staying. The tool vector was reverting. cert-manager was coming off; acme.sh was going back on. But acme.sh wasn't going to be doing what it had been doing before. It had been talking to ZeroSSL with default TXT placement. Now it would talk to Let's Encrypt with CNAME delegation.

Same tool. New work.

The mechanical work was smaller than that reframe implies. acme.sh was already installed. Its cron entry was already running — dead, from the failed authentication weeks earlier, but present. The credentials just needed rotation from Global API Key to API Token, already planned as part of the combined fix. The CA switch was one flag on the renewal command. And the challenge path needed pointing at the right place via --challenge-alias.

Or rather, needed to be pointed at the right place via --challenge-alias. The third obstacle was already waiting.

// third obstacle: the `--challenge-alias` discoverability gap

The third obstacle isn't quite fair to call a "failed attempt." It was more that the tooling had exactly the feature the situation needed, and the documentation didn't foreground it that way.

acme.sh supports DNS Alias Mode via the --challenge-alias flag. When you use CNAME delegation to redirect ACME challenges to a different zone — exactly the architecture the combined fix required — --challenge-alias tells acme.sh where to write the TXT record. Without it, acme.sh writes to the default location. With it, acme.sh writes to the aliased location. The flag is documented. The functionality works cleanly once configured.

But finding it required already knowing that's what you were looking for.

The acme.sh DNS Alias Mode wiki page describes the mechanism — "place TXT records at an aliased location via CNAME delegation." It does not describe the mechanism as "the solution to using base-zone NS delegation with a non-API DNS provider." Someone reading the DNS Alias Mode page top-to-bottom, without already knowing they need CNAME delegation for their specific problem, would come away understanding the mechanism but not connecting it to their situation.

I found it via a specific search for the problem, not by reading through the docs comprehensively. If I hadn't already been sitting with "I need to write TXT records at a location other than the default because SiteGround controls my authoritative nameservers and doesn't expose a DNS API," the wiki page would not have surfaced as relevant.

That's a documentation gap worth naming. It's not the same class of gap as missing documentation or wrong documentation. The docs exist. The docs are correct. The docs describe the mechanism accurately. What's missing is the connection between the mechanism and the class of problem it solves. Discoverability is a distinct axis from correctness, and the acme.sh DNS Alias Mode page is a case where correctness is high and discoverability is low.

Naming that gap publicly is more useful than pretending the answer was always obvious.

// the turning point

The turning point in the incident wasn't a moment of insight. It was a moment of reframing.

After the third obstacle, I sat down with the acme.sh docs again. Second read-through. Different question this time. The first read had asked "how do I do DNS-01 challenge with acme.sh?" — that had produced generic DNS-01 documentation, general-purpose, not obviously connected to the specific architecture I now had in place. The second read asked "what's the flag for the setup I've just configured?" — which is a much smaller, much more specific question.

The problem was framed correctly this time. So the answer was findable.

--challenge-alias surfaced immediately once I knew to look for the flag that told acme.sh "the TXT records go to a different zone than the domain's default authoritative one." That's the whole feature description in one line, and it maps directly to what the CNAME delegation was doing at the DNS layer.

What made this a turning point rather than "another thing to try" wasn't the flag itself. It was the shape of the fix collapsing at that moment.

The cert-manager path had been: diagnose the distributed bug across the zone-walker and the DNS layer, wait for the upstream patch, deploy the patched version, hope no new failures surface. Multi-day, multi-component, uncertain.

The acme.sh path was: add two CNAMEs at SiteGround, add one flag to an already-working cron entry, restart the cron, done. Single-digit hours, two-component, deterministic.

When the plan collapses from "diagnose distributed bug across three components" to "add a flag to an already-working tool," you know before you run it whether it's going to work. That's the anti-heroic version of the story. Not a moment of insight. A moment of reframing that made the answer obvious.

// the fix that landed

The mechanical fix was smaller than the diagnosis suggested it would be.

Two CNAMEs at SiteGround. _acme-challenge for the base domain pointing to a Cloudflare-hosted alias zone. The wildcard-challenge variant pointing to the same alias zone. SiteGround's DNS management supports CNAME creation via their control panel; no API required.

At Cloudflare, a dedicated zone for the alias, with a scoped API Token that had write access only to that zone. Not the domain's actual authoritative zone. Not the full account. Just the ACME challenge zone.

At acme.sh, the credential rotation from Global API Key to API Token in the account configuration. Two additions to the renewal command in the cron: --server letsencrypt to explicitly target Let's Encrypt (rather than acme.sh's ZeroSSL default), and --challenge-alias pointing at the delegated location. That's the whole acme.sh-side change.

At the Kubernetes layer, no change was needed. The renewed certificate would be dropped into the same Kubernetes secret acme.sh had been maintaining all along. The Gateway resource watching that secret would pick up the update automatically. Which is where NGINX Gateway Fabric earned its keep in this incident — but that's the next section.

The renewal ran clean the first time under the new configuration. acme.sh requested the challenge from Let's Encrypt, wrote the TXT record to the alias zone at Cloudflare via the scoped API Token, Let's Encrypt validated against the correct location (because CNAME delegation resolved correctly from Let's Encrypt's side), and the fresh wildcard cert issued. Total active work from "the flag surfaced" to "renewed cert deployed" was about half an hour — most of that spent waiting for SiteGround's NS delegation to propagate.

Two DNS records. Two flags. One credential rotation. And two vectors of the original migration (Let's Encrypt as CA, CNAME delegation as challenge architecture) preserved cleanly — cert-manager came off, but the harder-to-reverse changes stayed.

// NGINX Gateway Fabric added one thing for free

One small win worth naming, because it saved a step during the incident that would have otherwise been manual.

The certificate that acme.sh renews lives in a Kubernetes secret. The ingress layer that serves the certificate to inbound HTTPS traffic — in this stack, NGINX Gateway Fabric — reads that secret and terminates TLS with it.

Standard nginx-ingress-controller requires that when the secret changes, the ingress-controller pods either restart or be signalled to reload their configuration. Some deployments handle this via a sidecar that watches Kubernetes secrets and triggers reloads. Others handle it via periodic reload jobs. Some handle it by not handling it, and letting operators reload manually when they remember. All of those approaches add moving parts.

NGINX Gateway Fabric handles secret updates automatically. When the wildcard cert secret updated with the new acme.sh-issued certificate, the Gateway picked it up without any manual reload, without a sidecar, without a periodic job. TLS termination started using the new cert. No operator action required.

Not the story of this incident. But worth naming as a small win: the ingress layer was one of the few things in this whole stack that didn't need touching during the recovery. When something goes right during an incident, it's worth naming — especially when the same functionality in a sibling tool would have added another manual step under time pressure.

// what I'd make instinctive

Four lessons went into the runbook after this.

Cron exit codes need monitoring, not just cron execution. The renewal cron was scheduled correctly. It ran on schedule for weeks. Cron ran successfully — the shell command executed, cron logged the execution, the cron system considered the job complete. And every single execution was failing at authentication. Cron-ran-successfully monitoring is not the same as job-succeeded monitoring. Any critical automation — cert renewal, backup jobs, replication tasks, anything where "the cron ran" doesn't imply "the intended work happened" — needs both.

"Looks like working" is a distinct failure mode from "broken." cert-manager's zone-walking failure produced no error message, no failed status, no red dashboard indicator. Every log line read as healthy activity. The composition failed silently. That class of failure needs a diagnostic instinct closer to network engineering — read the actual traffic between components, don't trust either component's log about what it thinks it's doing. Error-reading skills don't apply when there are no errors.

Retry loops without circuit breakers can amplify already-loaded conditions. cert-manager's default retry behaviour was correct for transient failures and wrong for persistent ones. Twenty minutes of retrying against an already-loaded Cloudflare account was probably enough additional load to push the account into rate-limit rejection — not the sole cause, but a contribution the tool had no visibility into. Circuit breakers matter not just because they stop wasted work, but because they prevent the retry itself from becoming part of the failure. Know which of your automation has circuit-breaker behaviour and which doesn't.

Discoverability is a distinct axis from correctness. --challenge-alias works. Its documentation exists. What's missing is the connection between the mechanism the docs describe and the class of problem the mechanism solves. That gap turned a fifteen-minute config change into an hour of doc-reading. Documentation authors owe their readers not just correctness but discoverability — the framing that lets a reader find the right feature by describing their problem. Both are needed.

None of these are theoretical. Every one of them was learned by doing the diagnostic in the wrong order the first time.

// closing

Cert-manager saga posts tend to read as heroic sysadmin folklore. Someone stares at cryptic errors for days, has a moment of insight, cracks the case, writes it up as a triumph. The genre is well-established.

Most of them shouldn't be. The plans that collapse to a flag or a config change were never really the plans they looked like during the diagnostic phase. When the fix goes from "three-component distributed bug" to "two DNS records plus a config line," the diagnostic wasn't wrong — the frame was.

Reading this piece back, the whole incident could have ended on day one if the frame had been correct on day one. acme.sh was already in the stack. CNAME delegation was already a documented architecture pattern. --challenge-alias was already a supported flag. The pieces were all there. What wasn't there was the framing that connected them. cert-manager as the migration target was pulling attention toward the wrong tool for the specific problem the incident actually presented.

Article 8's GPU passthrough piece landed a similar-shaped observation about a different domain: the error pointed at the GPU, the culprit was the HBA. Different tools, different failure mode, same operator lesson. When the diagnostic is spending time in the wrong place, the frame is usually the problem — not the diagnostic.

If your renewal cron is running without monitored exit codes, that's the single highest-leverage thing you can fix this week. Not because it prevents this class of failure — but because it turns silent multi-week failures into loud immediate ones. Silent long failures are always worse than loud short ones. Fix the monitoring before you need it.

The error pointed at the GPU. The culprit was the HBA. — a Proxmox passthrough recovery

JJozzieTech — Thu, 20 Aug 2026 20:03:27 +0000

The plan looked textbook. PCIe passthrough an NVIDIA RTX A4000 from a Proxmox 8 host into an Ubuntu 24.04 VM, giving a Filecoin sealing worker direct GPU access for the compute phases that need it. VT-d on the Xeon. IOMMU enabled at the kernel line. OVMF BIOS on the VM. vfio-pci bound to the device IDs. Machine type set to q35 for PCIe support. Standard sequence, followed by every guide, ordered exactly as those guides ordered it.

Then qm start returned, and the VM did not come up.

Not a kernel panic. Not a host lockup. Not a "device not found" error surfaced to the operator. QEMU exited from the host's perspective and left nothing behind — no running guest, no console output, no obvious pointer at what had failed. The Proxmox web UI showed the VM as stopped, exactly as it had before I hit start.

The errors were there. They were just in a different terminal. dmesg -w on the host, running in a second window, produced a burst of IRQ allocation failures and vfio interrupt remapping errors scrolling past at VM startup, then silence. The host itself was completely fine throughout. Nothing was wrong with the box. Only the VM layer was broken, and it was broken in a way that read more like a misconfigured VM than a hardware conflict — which is exactly why the real cause took three sessions to identify.

The segmentation piece covered the network work that came before this. This piece is one of the workload-layer stories that followed — the GPU passthrough that took two evenings longer than it should have, because the error pointed at the GPU and the culprit was somewhere else entirely.

Two things this piece is not. It is not a step-by-step passthrough guide — there are plenty of those, and they are all fine until they aren't. It is the story of what happens when the checklist runs clean and the passthrough still refuses to work.

// what the GPU was actually there to do

Filecoin sealing — the process of turning raw storage into provable, verifiable data on the network — is not one workload. It's two compute phases with fundamentally different characteristics, and only one of them benefits from a GPU.

The first phase is SDR: Sequential Data Replication. It's designed to be resistant to parallelisation — memory-latency bound, CPU-heavy, and immune to GPU acceleration by design. On this hardware, SDR takes three to four hours per sector regardless of what else is in the box. No GPU speeds it up. That is intentional at the protocol level, not a limitation of the implementation.

The second phase is where the GPU matters. Tree building — specifically the TreeRC computations that produce the Merkle tree structure used in Proof-of-Replication — is GPU-eligible. Without acceleration, TreeRC runs on CPU and adds another two to three hours per sector. With the A4000 handling it, the same work completes in roughly fifteen to twenty minutes.

The framing that gets used in Filecoin marketing — "GPU acceleration makes sealing faster" — is technically true and operationally misleading. The GPU doesn't make sealing fast. It makes the second half of sealing fast. A storage provider without GPU acceleration is bottlenecked on tree computation as badly as on SDR, and the observable sealing throughput is roughly the SDR time plus the CPU tree time, sector by sector.

Where the GPU actually earns its place is in pipeline parallelism at scale. With the CPU pipeline running SDR on one sector while the GPU pipeline runs TreeRC on the previous one, sealing throughput becomes gated by SDR alone. That's the operator-relevant payoff — not making any single sector faster, but decoupling the two phases so they can run against different sectors concurrently. For a small storage provider, that is the difference between sealing two sectors a day and sealing five or six.

That was what the A4000 was there to do. Which is why the VM not coming up was a problem.

// the diagnostic that didn't converge

The investigation stretched across three sessions over about a week, mostly limited by the time I had rather than by the problem itself. Each session ran ninety minutes or so, and each one eliminated one correct-looking explanation without producing the actual cause.

Session one focused on the obvious: was the GPU actually bound to vfio-pci, and was the VM config correct?

lspci -nnk -d 10de:24b0

That output showed Kernel driver in use: vfio-pci — the host had surrendered the card, exactly as intended. No nouveau, no nvidia. The VM's PCI passthrough config referenced the correct BDF for the A4000, machine type was q35, OVMF was enabled, memory ballooning was off. Nothing in the config was wrong. That was session one.

Session two went down the driver blacklist and IOMMU group mapping path. /etc/modprobe.d/vfio.conf had the right device IDs listed. /etc/modprobe.d/blacklist.conf blocked nouveau, nvidia, nvidiafb, and snd_hda_intel. update-initramfs -u -k all had been run, and the host had been rebooted since. All correct.

Then the IOMMU group listing:

find /sys/kernel/iommu_groups/ -type l | sort -V

The A4000 at 03:00.0 appeared in one group with its HDMI audio companion at 03:00.1 — expected. But the same group also contained an IBM SAS2008 HBA. The HBA was managing the host's scratch storage, which meant it couldn't be passed through to the VM. And in an IOMMU group, all devices are the atomic unit of passthrough. You pass through the entire group or none of it.

That was where the pointer to the SAS2008 first appeared. But at that moment, I did not yet understand what shared-group membership specifically causes to fail. Group sharing is a widely documented passthrough concern, and the standard advice ("try the ACS override kernel patch") felt like a hack for a build that would need to run reliably long-term. I closed session two aware the SAS2008 was somehow involved but not yet certain how.

Session three was the crystallisation. Not a eureka moment — a slow narrowing. Once every other explanation was eliminated, the IOMMU group problem was the only candidate left. But the specific mechanism — why exactly a non-passthrough device in the same group breaks the passthrough of the target device — needed one more step of understanding before the physical fix made sense.

// FLR: what makes an IOMMU group problem actually a passthrough blocker

IOMMU group membership alone doesn't break passthrough. The Linux community's shorthand — "everything in the group has to be passed through together" — is correct but incomplete. It describes the safety constraint, not the failure mode. Understanding what actually goes wrong requires knowing what QEMU does at VM startup.

When the VM starts and takes ownership of a passed-through PCI device, QEMU has to reset it — bring it to a known state before the guest OS initialises. The reset mechanism it relies on is Function Level Reset (FLR), a PCIe capability that allows a device to be reset in isolation without affecting anything else on the bus. FLR is the clean, safe reset. It's what QEMU wants.

The A4000 supports FLR. The SAS2008 does not. And here is the mechanism: when QEMU tried to perform FLR on the A4000, it could not do so cleanly because the SAS2008 was in the same IOMMU group and could not itself be reset. The interrupt remapping infrastructure that isolates devices during passthrough could not correctly separate the GPU's IRQ handling from the HBA's, because from the IOMMU's perspective the two devices were treated as a single reset domain. The IRQ allocation failed. The VM didn't start.

The error pointed at the GPU because the GPU was the device QEMU was trying to bring up. The culprit was the HBA because the HBA was the device that couldn't cleanly get out of the way.

FLR support is directly checkable from the host. For any PCI device, the reset_method sysfs file lists the reset mechanisms the kernel considers available:

cat /sys/bus/pci/devices/0000:XX:XX.X/reset_method

For the SAS2008, that file did not list flr among its supported methods. That was the confirmation the mechanism I was reasoning about actually applied to this hardware. Once confirmed, the fix stopped being a question of ACS overrides or kernel patches. It became a question of physical topology.

This is the class of understanding most passthrough guides skip. They tell you what to configure. They don't tell you what to check when the configuration is right and the passthrough still fails. FLR-vs-non-FLR devices sharing an IOMMU group is the silent blocker at the heart of this failure mode.

// moving the HBA to a slot that mattered

The fix was physical. The SAS2008 had to move to a different PCIe slot — one that was not just physically distant from the A4000 but sat behind a genuinely separate PCIe root port. That distinction is critical and easy to miss.

Not every PCIe slot on a server board sits behind its own root port. Slot layout on the motherboard reflects physical trace routing, not IOMMU topology. Two adjacent slots may share a root port; two slots on opposite ends of the board may or may not. Moving the HBA one slot over on the same root complex would have done nothing — the IOMMU group membership would have been identical after reboot, and the failure would have repeated.

The target slot had to be chosen carefully. Two sources helped. The board's manual documents the PCIe topology — which slots connect to which CPU root complex, and how the lanes are allocated across the PCH. The existing IOMMU group listing also carries information: any two devices that had shown up in different groups before must, by definition, be behind different root ports. A slot known to be safe for the HBA was a slot whose currently-installed device (or its adjacent devices) had shown up in a group separate from the A4000's.

The physical work itself was straightforward. Full shutdown of the host — no live PCIe hotswap for this class of card. Chassis open. HBA out of its original slot. SAS cables temporarily disconnected (they had to be re-plugged after the move — a small operational risk if any cable connector had degraded, but nothing did). HBA into the target slot on the other root complex. Chassis closed. Boot.

Confidence going into that boot was moderate, not certain. PCIe slot physical location does not guarantee separate root complex membership — the only real proof is what the IOMMU group listing shows after boot. Moving the card is a testable hypothesis, not a guaranteed fix. If the target slot had turned out to be on the same root complex as the A4000 anyway, I would have been back at square one, chassis open again.

The fix confirmed itself in layers.

Structural first. After boot, the IOMMU group listing showed the SAS2008 in its own group, and the A4000 with only its audio companion at 03:00.1. That was the proof the physical move had produced the isolation it needed to produce. Before I even started the VM, I knew the group problem was resolved at the topology layer.

Functional second. VM started. QEMU didn't exit. Guest OS came up. nvidia-smi inside the VM reported the A4000 with its full 16 GB of memory and no error state. CUDA initialised. The card was talking to its driver.

Ecological third. The sealing worker initialised, picked up a queued sector, and the first GPU-accelerated TreeRC task completed in fifteen minutes instead of three hours. That was the confirmation the whole endeavour had been about.

Three checks, each validating a different layer, each necessary. The structural check proved the topology; the functional check proved the driver stack; the ecological check proved the workload actually benefited. Skipping any one of them would have left the fix incompletely verified.

// the second time was clean

The Filecoin project was later scrapped for reasons unrelated to the GPU passthrough — project constraints outside this story. The A4000 came back into the general hardware pool. When MMX came online later as a farming workload, the passthrough pattern needed to run again on a different guest OS.

MMX is a Chia fork with an important twist relevant to storage layout: it supports both HDD plots and SSD plots, and the SSD plots are compressed. Farming compressed plots requires GPU-accelerated decompression at every proof challenge — the harvester has to expand the plot data on the fly to check whether it holds a winning proof. Without a GPU, compressed SSD plotting is effectively non-viable at scale. With one, the decompression happens fast enough that the harvester keeps up with challenges in real time.

MMX and Chia have always run on Windows VMs on this network — a convention that predates this build, driven purely by the maturity of the Windows GUI tooling around Chia's ecosystem rather than any technical requirement. So the second passthrough build targeted a Windows guest instead of Ubuntu.

The build was clean. IOMMU group listing checked before touching physical hardware — target slot known-safe. Device IDs identified. VFIO binding confirmed. Windows installed with only the emulated VGA present, drivers applied after boot, GPU attached in a second pass to avoid the display blackout risk when the NVIDIA driver takes over primary output. Two extra CPU flags on the VM: hidden=1 to mask the hypervisor and avoid the notorious Error 43, and +pcid for TLB performance under memory-heavy workloads.

One clean build. No gotchas. Roughly one session, no diagnostic sessions after.

The pattern only reads as clean because the checks became routine. That is what the Filecoin project's expensive lesson bought.

// what I'd make instinctive

Four things went into the runbook after this.

Check IOMMU groups before touching physical hardware. The listing takes seconds to produce and tells you exactly what the passthrough constraints will be before you start. Every hour of build effort saved by checking the group listing first is an hour that doesn't need to be spent later, chassis open, moving cards under time pressure.

Non-FLR devices in the same group are silent blockers. IOMMU group membership on its own doesn't fail the passthrough. It's the combination of shared group and one device without FLR support that produces the specific failure mode that took me three sessions to identify. If your target device is in a group with a device that doesn't support FLR, that group needs to be broken up before passthrough will work. The reset_method sysfs check is thirty seconds and catches this class of problem definitively.

Physical slot choice is a topology question, not a proximity question. Adjacent slots may or may not sit behind separate root ports. Distant slots may or may not either. The board manual and the existing IOMMU group listing between them tell you which slots are actually independent. Guessing based on physical distance is worse than guessing at random, because it feels like a plan.

Layered confirmation matters — structural, functional, ecological. A working passthrough is not the same as a working workload. The structural check (IOMMU group listing) proves the topology. The functional check (nvidia-smi or equivalent) proves the driver stack. The ecological check (the actual workload running against real work) proves the whole system does what it was built to do. Skip any layer and the fix is only partly verified.

None of these are theoretical. Every one of them was learned by getting the diagnostic order wrong the first time.

// closing

The GPU passthrough pattern extends far beyond either of the workloads that surfaced in this piece. Any hardware acceleration that a VM needs — an AI or ML lab running CUDA training jobs, a transcoding server that needs hardware video encoders, a gaming VM that needs a discrete GPU, or a media server that needs GPU-accelerated encoding for streaming to remote clients — hits exactly the same setup questions, the same IOMMU group constraints, and the same class of neighbour-check that this piece is built around.

The unified lesson from this piece is small and load-bearing: the error pointed at the GPU. The culprit was the HBA. Check the neighbours before you blame the card.

If you're planning GPU passthrough on a Proxmox host, do this before you touch any physical hardware. Enumerate the IOMMU groups on the current configuration. Identify what shares a group with your target device. Check the reset_method sysfs entry for each of those neighbours. If any of them lacks FLR support, plan the physical move now — not later, when the VM refuses to start and the error message is pointing everywhere except at the actual problem.

The checklist runs clean. Until it doesn't. This is the check that catches when it doesn't.

Recovering a ZFS pool with a missing RAIDZ1 vdev — without wiping a single disk

JJozzieTech — Thu, 13 Aug 2026 20:25:55 +0000

The same outage that suspended the first pool also took down a second pool on the same box in a different, harder way. That's the piece the suspended-pool recovery closed by pointing at. This is that piece.

The first pool showed multiple disks marked as faulted, but the vdev structure was intact. It was a path issue that looked catastrophic. The second pool, on the same box, hit by the same outage, showed something worse: an entire RAIDZ1 vdev in the UNAVAIL state, with every disk in that vdev marked as REMOVED. The other vdev in the pool was fully ONLINE. But that didn't matter.

A ZFS pool striped across multiple vdevs is only as available as its least-available vdev. If one vdev is unavailable, the pool is unavailable, even if every other vdev is healthy. Pool-level parity does not protect against a whole vdev going away — that's not what ZFS's striping model does. If you want the pool to survive one vdev disappearing, you need mirror-of-vdevs or dRAID or another topology that plans for that specific failure. RAIDZ1 across vdev-striping does not.

I refer to the affected pool as MediaPool-B, matching the naming from the sibling piece.

The recovery for this pool was fundamentally different from the first one's. The first pool's recovery was zpool clear after checking the disks were visible. This pool's recovery required reading the ZFS labels directly to confirm the "removed" disks still had valid pool metadata — before running any command that could touch state. And after the pool came back online, there was a second problem waiting: several hundred files with permanent errors that had to be handled cleanly before the pool was truly clean again.

// what the console showed

The pool's status output, structurally:

MediaPool-B  SUSPENDED
  raidz1-0   UNAVAIL
    disk 1   REMOVED
    disk 2   REMOVED
    disk 3   REMOVED
    disk 4   REMOVED
  raidz1-1   ONLINE
    disk 1   ONLINE
    disk 2   ONLINE
    disk 3   ONLINE
    disk 4   ONLINE
    disk 5   ONLINE
    disk 6   ONLINE

Two RAIDZ1 vdevs in the pool. One fully online. One showing every disk as REMOVED and the vdev itself as UNAVAIL. The pool as a whole in SUSPENDED state.

The visual read of that output is that half the pool has evaporated. The instinct is to look at the TrueNAS GUI, see disks appearing as "unassigned," and reach for either "Add to Pool" or "Wipe." Both are wrong. Both are, in fact, the most dangerous things you can do in this situation.

The reason is a distinction the TrueNAS SCALE GUI doesn't surface clearly. A disk shown as "unassigned" in the GUI is a disk that TrueNAS is not currently associating with a pool through its own tracking. That's not the same as a disk without a ZFS label on it. Every disk that was ever a valid ZFS pool member has labels on it — four of them, redundantly placed at the start and end of the device — describing the pool it belongs to, the vdev it was part of, and the identifier that makes it a member. Those labels survive the pool being suspended. They survive TrueNAS forgetting about the pool. They survive a reboot. What they don't survive is you wiping the disk, or adding the disk to a new pool structure. Both destroy the labels. Neither is recoverable.

So the first correct move is exactly the one the GUI is designed to push you away from: don't touch the disks yet.

// the critical move: don't trust the GUI

The right move is to read the ZFS labels directly, before any GUI action, to prove those "unassigned" disks are still legitimate members of the pool that ZFS thinks is broken. The tool is zdb, ZFS's low-level debugger, and the flag is -l for label inspection.

The pattern I used was a shell loop that scanned every partition on every disk, called zdb -l on each, checked whether the label mentioned the pool by name, and printed the relevant identifying fields for any match:

for d in /dev/sd*2; do
  zdb -l "$d" 2>/dev/null | grep -q "MediaPool-B" && \
    echo "===== $d =====" && \
    zdb -l "$d" 2>/dev/null | egrep "name:|guid:|top_guid|path:"
done

The output came back with entries for the four disks that had been shown as REMOVED. Each entry had a name: field matching MediaPool-B, a guid: uniquely identifying that member disk, and a top_guid: field identifying which vdev the disk belonged to. The four top_guid: values matched each other — meaning the four disks did all belong to the same vdev — and matched the top-vdev GUID of the missing raidz1-0.

That was the turning point of the recovery. The four disks were not gone. They were not damaged. They were not the wrong disks. They were exactly the four members of the missing vdev that they'd always been, still present with their labels intact, waiting for ZFS to resume the pool.

This is the class of check that matters at the moment when the GUI's information is missing the shape of the actual state. zdb -l doesn't care what TrueNAS thinks the disks are for. It reads what's actually written on the disk. That distinction — between what a management layer says about a disk and what the disk itself says about its history — is the whole point of ZFS's label design. Nothing at the storage layer should trust the management layer's model of state over the storage layer's own record.

If any of the four disks had failed to show a matching label, or had shown a label for a different pool, the response would have been different. In that case, the correct move would be to stop, escalate to the physical layer, and only come back to software recovery after understanding why a disk was reporting inconsistently. Nothing was inconsistent. Every disk reported exactly the label expected. The pool was recoverable.

// zpool clear, this time with confirmation

With the four missing disks confirmed as legitimate vdev members via their labels, the recovery command was the same one the first pool's recovery used:

zpool clear MediaPool-B
zpool status -v MediaPool-B

The pool came back to ONLINE. All disks online. READ, WRITE, and CKSUM counters at zero across every device. The vdev that had been UNAVAIL was now healthy. The pool was importable, mountable, and — critically — the labels told ZFS the same story the labels had always told, so the pool's history and identity carried through cleanly.

This is where the confidence delta from the first pool's recovery mattered. In the first pool, zpool clear was applied after confirming the disks were visible and the kernel path was quiet. Here, it was applied after confirming the disks weren't just visible — they were still legitimate members of the pool ZFS was trying to bring back. Different level of certainty, different level of stakes. A blind zpool clear on the first pool's shape would have worked. A blind zpool clear on this pool's shape, without the label check first, would have been a bet on state that hadn't been verified.

But zpool status after the clear showed something the first pool didn't produce.

// the second problem: permanent errors

The pool was ONLINE. Every disk was ONLINE. Every counter was zero. And ZFS was reporting permanent errors in application data — specifically, several hundred files that had errors ZFS couldn't reconcile from parity.

Permanent errors mean exactly what they sound like. ZFS has tried to reconstruct these files from the parity blocks in the vdev, and the reconstruction can't produce the file the pool's metadata says should exist. The file's content is unrecoverable at the ZFS layer. What you do next depends entirely on what the files are.

For this recovery, the affected files were regeneratable workload data — application-layer files that were expensive to reproduce but not irreplaceable. If they'd been irreplaceable — family photos, business documents, source code without a git remote — the correct next step would have been to restore from backup for those specific paths, then clear the error state. Application context determines the correct operational response. There is no single answer.

The sequence for regeneratable data:

Save the evidence first. The zpool status -v output lists every affected file by path, and that list is the ground truth of what's actually corrupt. It should be captured to disk before anything is deleted, so the record survives whatever comes next:

zpool status -v MediaPool-B > /root/MediaPool-B_corrupt_files_$(date +%F_%H%M).txt

Extract just the file paths, filtered to the affected dataset:

zpool status -v MediaPool-B \
  | grep "/mnt/MediaPool-B/Dataset/" \
  > /root/MediaPool-B_corrupt_delete_list.txt

Count and preview before doing anything destructive:

wc -l /root/MediaPool-B_corrupt_delete_list.txt
head -20 /root/MediaPool-B_corrupt_delete_list.txt

In this case, the delete list came out to 311 files, every one of them under the expected dataset path. That confirmed the impact was fully contained to the regeneratable workload before removing anything.

Delete the corrupted files with whitespace trimming — the zpool status output leaves leading spaces on file paths, and rm doesn't handle those well:

while read f; do
  f="$(echo "$f" | xargs)"
  rm -v "$f"
done < /root/MediaPool-B_corrupt_delete_list.txt

Then verify none of the listed paths still exist:

while read f; do
  f="$(echo "$f" | xargs)"
  [ -e "$f" ] && echo "$f"
done < /root/MediaPool-B_corrupt_delete_list.txt | wc -l

Zero. All 311 files gone.

Clear the error state and scrub:

zpool clear MediaPool-B
zpool scrub MediaPool-B

The scrub is what actually retires the stale error header. zpool status immediately after zpool clear will still show the permanent-errors block with no files listed — the header persists until a scrub confirms all data is clean. After scrub completed, the pool status came back to:

MediaPool-B = ONLINE
scrub completed with 0 errors
errors = No known data errors

// what SMART said afterward

Before declaring the recovery complete, I checked SMART health on each of the four disks that had been marked REMOVED. Any of them could have had a real physical issue that contributed to the outage's severity on this pool.

smartctl -a /dev/diskA
smartctl -a /dev/diskB
smartctl -a /dev/diskC
smartctl -a /dev/diskD

Three of the four came back clean — SMART health OK, normal temperatures, zero uncorrected errors, zero reallocated sectors. Fourth disk came back with a small grown-defect count. Not zero, but low enough that immediate replacement wasn't the answer. That disk went on a watch list to be checked at each subsequent scrub, with the plan to pull it if the count grew.

The lesson here is a small one but worth stating: recovery doesn't end at "the pool is back." It ends at "the pool is back and the drives that were involved in the incident have been individually checked for physical health." Neither one substitutes for the other.

// what I'd make instinctive

Four things went into the runbook after this pool.

Unassigned in the GUI is not the same as unused. TrueNAS SCALE shows disks as "unassigned" when its own tracking doesn't currently associate them with a pool. That has almost nothing to do with what's on the disk. A disk that's shown as unassigned can still hold valid ZFS labels for a pool the system is trying to bring back. The GUI's model of state is separate from the storage layer's model of state. When those diverge, trust the storage layer.

Read ZFS labels before doing anything destructive. zdb -l is cheap to run, always available, and reads exactly what's written to the disk. It doesn't rely on any pool being imported, any middleware being healthy, or any GUI showing correct information. When something looks catastrophically wrong with a vdev, zdb -l is the first tool to reach for, not the last. Wiping a disk that still has valid pool labels is destructive and not recoverable — thirty seconds of label inspection prevents that class of mistake entirely.

One unavailable vdev takes the whole pool down. If the pool is striped across multiple vdevs — the default for capacity-oriented ZFS deployments — the pool's availability is the availability of its worst vdev. Mirror-of-vdev or dRAID topologies are the alternatives when one whole vdev going away isn't an acceptable failure mode. Choose the topology deliberately, and know what failure modes you've bought into.

zpool clear doesn't guarantee no permanent errors. The pool coming back online means ZFS has decided the vdev structure is workable. It doesn't mean every file survived. Permanent-error handling is a separate operational step, and the correct response depends entirely on whether the affected files can be regenerated or need to be restored from backup. There is no single answer that's right for every dataset.

None of these are theoretical. Every one of them was learned by doing the work right the first time — or nearly wrong once, which counts for the same lesson.

// closing

Two pools, two recoveries, one outage. The first pool was a path issue that looked like a shelf failure — zpool clear after confirming the disks were visible. The second pool was a vdev issue that looked like half the pool had evaporated — zpool clear after confirming the labels were still valid, then several hundred files deleted, then a scrub. Different failure modes, different recovery sequences, different mental models. Both pools came back without any physical replacement.

The unified lesson across both pieces is small but load-bearing: ZFS's failure reporting is designed to be interpreted, not complied with. The console output tells you what ZFS observed. It doesn't tell you what actually happened, and it doesn't tell you what to do about it. That interpretation is the operator's job, and it depends on understanding what the storage layer is actually asking of you at that moment.

The ongoing storage roadmap for this rack starts with backup posture — the PBS project referenced in the CORE-and-SCALE piece is what turns "recovered without backup" from operator luck into operator design. Recoveries this clean shouldn't have depended on the outage being transient. That's a gap I'm closing, not a story I'm telling as a success.

If you're running mixed-workload TrueNAS today and haven't tested a backup restore recently, that's your homework. It's mine too.

After a TrueNAS ISO reinstall — the two things that broke that no one warned me about

JJozzieTech — Thu, 06 Aug 2026 19:57:14 +0000

The TrueNAS SCALE machine that runs my SMB shares was in trouble. Windows clients had stopped seeing the shares reliably — intermittent at first, then completely. When I dropped to the shell to check, systemctl status smb returned something worse than a failed service: it returned a service that no longer existed. Same for nmb. Same for winbind. The service wrappers were gone.

Samba's binaries were still there. The actual smbd and nmbd executables under /usr/sbin were untouched. But the systemd units that normally wrap them, and the TrueNAS middleware that manages both, had lost too much of themselves to be repaired by unit regeneration or middleware restart. Something at the service-layer wiring had broken beyond in-place fix.

The recovery answer, once I stopped trying to patch it, was a fresh ISO reinstall and a pool reimport. That got the service layer back. What it didn't do — what nobody had warned me it wouldn't do — was fix Windows authentication or fix ACL permissions on the imported datasets. Two things I'd inherited from the reinstall broke SMB access in ways that had nothing to do with SMB itself.

This piece is the story of those two things. The SID mismatch that made Windows reject correct credentials. The missing default ACL entries that blocked recursive permission repair. And the fix sequence that restored SMB access to what it had been before the service layer disappeared.

Same box as the ZFS suspended pool piece, different failure mode. When the storage layer goes wrong, ZFS suspends the pool and tells you exactly that. When the service layer goes wrong, the layer just disappears and Windows quietly stops answering.

// why the fresh ISO was the answer

The service layer on TrueNAS SCALE isn't just Samba. It's the systemd units that manage Samba. The middleware plugins that TrueNAS uses to expose SMB configuration through its GUI. The internal API endpoints that the middleware calls when you edit a share in the interface. All three layers were compromised.

I tried the standard recovery moves first. Regenerating service units through the middleware. Reloading systemd. Restarting middlewared to force the internal API to reinitialise. None of it took. The problem wasn't a single missing piece — it was structural damage across the middleware, systemd, and Samba integration layers that had drifted enough that regeneration couldn't reconstruct working state from what remained.

That's the moment where the answer stops being surgery and starts being a reinstall. Not the emotional decision — the reasonable one. When a service layer's structural integrity is gone, patching pieces individually just produces a service layer that limps. A fresh ISO reinstalls all three layers cleanly, at their intended versions, in their intended configuration. Fifteen minutes of install, a pool reimport, and the service layer is back to what it should be.

That's what happened. Fresh ISO. Reimport of pools. smb.service, nmb.service, winbind.service all back. midclt smb.status returning cleanly. The GUI managing SMB again without errors.

Then the actual story of this piece started.

// the two things that break after reinstall

With the service layer working again, Windows still couldn't authenticate to the shares. Correct credentials, rejected by every share on every dataset. The problem wasn't SMB anymore.

The SID mismatch.

When TrueNAS SCALE reinstalls, every user account has to be recreated. Same usernames, same passwords, same group memberships — nothing prevents you from recreating them exactly. But Windows doesn't care about your username. What Windows cares about is the SID.

A SID is the Security Identifier that Windows uses to actually attach permissions to identities. Usernames are human-readable labels layered on top; the identifier that gets written into ACL entries is the SID. When TrueNAS creates a user, it generates a SID. When TrueNAS is reinstalled and the same user is recreated, a new SID gets generated. Not the same one. The SID is not derived from the username.

Existing ACL entries on the datasets — the ones from before the reinstall — reference the old SID. Windows looks at those ACL entries, matches its authentication attempt against the SID recorded there, sees no match against the new user's new SID, and rejects the authentication with "Credentials do not match."

The credentials do match. The SIDs don't. Windows is telling the truth in a way that doesn't help you fix it.

The missing default ACL entries.

The second problem shows up when you try to fix the first one by applying corrected permissions recursively. TrueNAS SCALE's ACL editor supports recursive apply — set permissions on a parent dataset, tell it to apply down through everything below, and it walks the tree. That's the normal repair for permission drift.

Recursive apply on imported datasets errors out immediately:

Error: dacl -- Default ACL entries are required in order to apply ACL recursively

That message is opaque unless you know what a default ACL entry is. Access ACL entries govern the files and directories that exist right now. Default ACL entries govern what permissions get inherited by new files and directories created inside a parent. On a healthy TrueNAS dataset both are set. On a dataset imported from a pool after reinstall, only the access entries survived the import — the default entries didn't come across.

Without default entries, recursive apply doesn't know what to inherit. It refuses to write any of the changes rather than apply them incompletely.

// the fix sequence

Once both problems were understood, the recovery ran in a specific order.

Step one: establish default ACL entries.

Recursive apply had to work before anything else was possible. Establishing default ACL entries on the affected datasets was the unblock.

TrueNAS SCALE ships preset ACL configurations that include both access and default entries. The RESTRICTED preset is the conservative choice — root and the dataset owner get full control, everything else is denied by default. Applying it to the dataset writes both access entries and default entries in the correct form, then recursive apply works.

Step two: recreate the SMB user with the intended group memberships.

The SMB user recreation was straightforward through the TrueNAS GUI. Same username, same password, same primary group. The new user got its new SID assigned automatically. Nothing about this step required anything unusual — the account creation is standard TrueNAS SCALE user management.

The systemd and middleware refresh sequence run after the reinstall to confirm the service layer was clean:

midclt call service.generate_all_units
systemctl daemon-reload
systemctl restart middlewared

Those weren't necessary for the ACL fix — the fresh ISO had already given us a clean service layer — but they're the standard TrueNAS SCALE service-layer refresh sequence, and running them confirmed no residual issues before the ACL work started.

Step three: apply corrected permissions recursively.

With default ACL entries in place from step one, and the SMB user existing with a new SID, the corrected ACL now needed to be written down through the dataset tree. The RESTRICTED preset had already established the ACL structure; adding the SMB user with full control on the dataset and applying recursively wrote the correct SID into every ACL entry through the tree.

Step four: verify from Windows.

Windows client connected to a share, authenticated with the SMB user's credentials, got in cleanly. No more "Credentials do not match." The credentials the user was typing had always been correct — they now matched an ACL entry whose SID pointed at the user actually trying to authenticate.

The ACL work took the same session as the reinstall itself. Most of the time was working through the preset and recursive apply through several affected datasets. The recursive apply itself was fast once it stopped erroring.

// what I'd make instinctive

Three things went into the runbook after this.

Snapshot the system config before major changes. TrueNAS SCALE lets you export the entire system config to a downloadable file — settings, users, ACL definitions, share configurations. Before this incident, I hadn't done it. If I'd had a config snapshot from before the SMB service layer failed, the fresh ISO reinstall would have restored not just the service layer but also the user accounts with their original SIDs. The SID mismatch problem wouldn't have existed. That single missing habit was the whole reason step two of the fix took the form it did. Config snapshots now go with every major change to the box.

Pool imports don't recreate everything the datasets need. Default ACL entries are the visible case; they're the one that surfaces an error message. But the principle is broader. When you import a pool into a fresh TrueNAS installation, you get the pool structure and the actual file data — you don't automatically get all the metadata state that the original installation had built up around the datasets. Some of that state has to be rebuilt manually. Assume it, don't be surprised by it.

Know the SID story before you need it. SIDs are how Windows actually attaches identities to permissions, and username is a display convention. Reinstalls change SIDs. Any ACL entry pointing at an old SID will silently fail to match a new user of the same name. The "Credentials do not match" error is misleading because the credentials aren't the problem. The identifier attached to the credentials is.

None of these are theoretical. Every one of them was learned by doing the work slightly wrong the first time.

// closing

This piece is about the layer between storage and client — where user identity meets file permissions, where authentication decisions happen. When that layer breaks, the storage is still fine, the network is still fine, and the client is confused. The SMB failure category is different from the ZFS failure category. Different response, different recovery, different mental model.

The next piece in this series is another storage-layer story — a ZFS pool where an entire RAIDZ1 vdev appeared to be gone, and the recovery required reading ZFS labels directly to confirm the disks were still valid pool members before touching anything. Different layer, different failure mode, different set of things that turned out not to be as broken as they looked.

If you're running TrueNAS SCALE and haven't exported a system config recently, do it now. That's the single highest-leverage thing you can take from this piece. The config snapshot is what turns a fresh ISO reinstall from a story about SID mismatches into a story about ten minutes of downtime.

A Kava validator on my existing stack — how the plan came together, and what the second chain will do differently

JJozzieTech — Thu, 30 Jul 2026 20:36:36 +0000

The plan didn't start at Kava. It started at the Cosmos Hub, because the flagship chain is where you go if you're not sure where to go.

The Cosmos Hub's active set — the top validators eligible for block rewards — has a floor. To enter it, a validator has to hold more staked ATOM than the smallest active validator does. In mid-2026 that floor sits at tens of thousands of ATOM — hundreds of thousands of dollars at current prices. That's an infrastructure business, not a homelab experiment. It's the size of stake where you're raising capital before you're writing a .toml file.

I don't have that capital, and I wasn't going to raise it to prove a point about self-stake. So the plan reshaped itself. Instead of validating on the flagship, target a smaller Cosmos appchain — one where a modest self-stake actually gets into the active set, but the chain is real enough that the validator work isn't performative. Kava won selection as chain #1 on that criterion. Real DeFi product with actual users, Binance-listed, healthy market cap, an active set that was reachable for a modest self-stake at the time I entered. Enough consequence to make the work matter; enough reachability to make the work possible.

Every validator writeup I've ever read assumes you got past this decision on money and skips to the software. This one starts here because for most people evaluating whether to run a validator, this is where the plan actually ends. I want the piece where it doesn't.

One thing the software will not forgive: the same cryptographic key that earns the validator's rewards will, if it ever double-signs a block, get the operator permanently and irrecoverably slashed. Not a warning. Not a temporary jail. The stake and any bonded delegator stake gets burned by the chain itself. Every design decision in what follows is downstream of that constraint.

// the hardware happened to have the right shape

The T430 wasn't acquired for this. It was already in the rack, sitting mostly idle, repurposed from a Filecoin project that didn't pan out. 512 GB of ECC on the box total, most of it committed to what had been the Filecoin VM, and the validator was going to draw from the balance. Because the RAM was there, the sizing wasn't a scarcity negotiation. There was room to be comfortable rather than minimal.

The validator VM got 8 vCPU and 48 GiB. Kava's documented minimums are lower than that. The provision deliberately sat above the documented floor rather than at it, because the downside of under-provisioning a signing node is asymmetric. A resource-starved node misses blocks. Missed blocks eventually jail the validator. Jailed validators lose bonded stake. There is no upside to a lean signing node worth the risk of a starved one, so it got comfortable margin.

The 48 GiB is a fixed allocation. Balloon disabled. Deliberate: the validator's RAM is reserved, not dynamically reclaimable, so memory contention with any other VM on the same host can't starve it. That decision followed the same pattern the Filecoin VM used, which had proven itself.

Storage: a 3.5 TB SSD, passed through to the VM as a dedicated device, mounted at /data/cosmos. Not carved from the shared LVM pool that holds VM OS disks. That separation was an explicit design choice — OS-disk failure and chain-data failure are two independent things that shouldn't share a failure mode. If the OS disk dies, the chain data survives. If the chain data corrupts, the OS is untouched. Two failures stay two failures.

The IOPS discipline was verify-before-trust. Before the node was trusted with signing, the disk was benchmarked: fsync p99 around 734 microseconds, roughly 5,353 IOPS on the workload that mattered. Adequate for the validator's write pattern, confirmed by measurement rather than assumed by spec-sheet reading. The chain is currently running lean on that volume — around 48 GB used of 3.5 TB available — so the storage layer is under no near-term pressure.

That's the hardware. Nothing about it was a targeted purchase for validating. The box had the right shape because the rack had the right box, and the decisions that made the box right for the previous project turned out to make it right for this one too.

// the snapshot that wouldn't replay

The first real war story landed at the snapshot restore step, before signing had ever happened.

Full sync from genesis wasn't the option — the chain history is large enough that starting from block zero would have run the node for days before it caught up. Snapshot restore is the standard alternative: download a recent state snapshot from a public provider, extract it to the chain data directory, start the node from there. Fifteen minutes of work, then the node picks up from a recent height and catches to tip.

The first snapshot source I tried, Polkachu, produced state that wouldn't replay cleanly. The node started, entered replay, and choked partway through with IAVL state that wouldn't reconcile under standard pruning parameters. Three attempts. Each one an evening. Each one ending the same way.

The fix was switching to a different snapshot source — NodeStake — and the replay went clean the first time. Nothing wrong with Polkachu as a provider; the specific snapshot that day just had state my node couldn't reconcile.

What came out of the pain became doctrine: verify the canonical AppHash from a public RPC before committing to a long snapshot mount. Fifteen seconds of check. If the snapshot's AppHash doesn't match what the chain reports at that height, the snapshot is bad — save yourself the hours. That check now runs before any snapshot restore on any chain in the pipeline. It's a five-line addition to a preflight script that would have saved me three evenings on this build.

// Horcrux 2-of-3: the signing architecture

Threshold signing is where the double-sign constraint gets solved.

Horcrux takes the validator's private key and splits it mathematically into shares. Each share, on its own, is useless — it can't sign anything. To produce a valid signature, a configurable threshold of shares must cooperate. The private key never exists as a whole thing on any machine; it only exists as an assembled signature at the moment it signs. There is no file on any host that, if stolen, gives an attacker a working key.

The topology choice is 2-of-3: three cosigners, any two of which are enough to sign. That specific shape is the homelab-scale sweet spot. It buys two properties simultaneously.

Fault tolerance. Any one cosigner can go down — for maintenance, a crash, a host failure — and signing continues on the remaining two. No single machine is a single point of failure for the whole validator.

No single point of compromise. No one machine ever holds a usable key. An attacker who takes one cosigner gets one share, which is worthless without a second. The double-sign threat is neutralised at the architectural level.

Alternative topologies each fail one of those tests. 2-of-2 has no fault tolerance — lose one, signing stops. 3-of-5 buys more redundancy than a homelab cluster with a handful of hosts needs while widening the surface area an attacker has to work through. For a small cluster with a no-slashing-tolerance constraint, 2-of-3 is the minimum topology that gives both properties. That's the honest reason it's the standard homelab-scale answer.

Physical distribution matters as much as the math. The three cosigners run on pro1, pro2, and pro4. The validator's full node runs on pro5. The standby node runs on pro3. No host runs both a cosigner share and a signing node. If any one physical host dies, you lose at most one share — not one share and the node together. Correlated failures at the host level get decorrelated by placement.

The honest nuance: whether every original placement decision was made purely for that reason or partly reflected which hosts had capacity, I can't fully reconstruct. What I can say is that the principle is deliberate — cosigner hosts should be separate from node hosts — and the same principle got applied more explicitly when the second chain (Akash) came online later. It reads as doctrine, and it functions as doctrine, even if the Kava-era placement was principle plus fit rather than principle alone.

The Kava-specific setup went cleanly. The shares distributed on the first attempt, the cosigners came up in the intended order, and the Horcrux Raft cluster formed quorum without incident. The hard-won signing lessons — the startup-ordering rule, the empty-placeholder-key-file gotcha, the strict host-separation rule — surfaced later on the Akash build. Those live in a future piece, not this one.

// the last check before bonding

Before the validator was ever bonded to the network, the check that mattered most was the one that proved Horcrux was actually doing the signing.

The mechanics: Kava's validator identity is normally represented by a file called priv_validator_key.json on the full node. It contains the raw private key the node uses to sign blocks. When Horcrux is doing the signing instead, that file must not be present on the node — otherwise the node would sign locally with its own key, which defeats the entire threshold-signing architecture and reintroduces the single-key-compromise risk.

The discipline is this: archive priv_validator_key.json before going live. Confirm the node is signing through Horcrux with no local key present. Only then bond stake to the validator. Only after signing-under-bond is empirically confirmed — real signed blocks recorded on the chain, produced by the threshold-signing cluster with no local key on the node — is the archived key destroyed.

The reason for that specific ordering is asymmetric downside again. If Horcrux fails silently and the node is still capable of falling back to a local key, the double-sign protection is theoretical rather than real, and the operator won't find out until it matters. If the local key is confirmed absent and the node signs anyway, the protection is real. If the local key is confirmed absent and the node stops signing, that's a fast, safe, revertible failure — you archive-restore the key, diagnose Horcrux, try again. Nobody's stake is at risk during the check.

That check is what "the double-sign will get you permanently slashed" reduces to, operationally. Everything before it is preparation. Everything after it is delegator-visible.

// first signed block, first delegator

The moment the validator was bonded and started signing, I didn't notice.

That's how it's supposed to go. When you've verified everything before going live — the storage, the sizing, the snapshot, the threshold-signing, the local-key discipline — the moment of going live is a non-event. The chain assigns a block. The validator signs it. Then another. Then another. There is no cinematic moment. There is a voting_power field on the node's /status endpoint that reads as a nonzero number instead of zero, and there are heights ticking upward in a monitoring panel, and the operator does not notice because nothing has broken.

That's success. Cinematic first-signed-block moments belong to validators that went live without checking. The absence of drama is what the pre-flight work was for.

The emotional beat, when it did arrive, was three days later. A stranger — someone I had no connection to, no relationship with, no reason to expect — looked at the validator, made an evaluation, and delegated stake to it. First independent delegator. Two total on the validator now, mine plus this one. That's the moment the whole thing stopped being an experiment and became a service someone was using.

Here's the current state, as of writing:

D+28 operational. Uptime 100%. Window block miss: 0 out of 10,000. Two delegators. That's not brag material — it's the operational state, the same way zpool status in the ZFS recovery piece was the state. Numbers on a public chain, verifiable by anyone. The point isn't the numbers. It's that this is what the pre-flight work produces.

// building the safety net: standby + drilled failover

The single node was the launch state. The current state is different.

There's now a standby node on pro3 — a second full node, kept synced to chain tip, ready to take over signing if the primary goes down. Same chain data, same signing configuration, same Horcrux endpoints. The switch from primary to standby is scripted — one script triggered by the operator with a confirmation prompt, another script to fail back to primary once it's healthy again.

The manual failover benchmark, from an earlier phase before automation, was 12 minutes 55 seconds from primary-down to signing-resumed. Human-driven. Every step verified by hand. Correct, but slow.

The scripted failover, drilled against a real deliberately-triggered outage — not a simulation, an actual systemctl stop kava on the primary — clocked at 1 minute 49.5 seconds from primary-down to signing-resumed. 37 blocks missed against the chain's 10,000-block jailing window. Roughly 0.4% of the jailing buffer consumed for a real failover event. Big enough to matter; small enough to survive.

The interesting part of the drill wasn't the failover script — it was the failback. The failback script's first dry run — the one designed to bring primary back into service after standby had taken over — surfaced two real bugs that had never been noticed. A missing privval-blank step, which would have hung real incidents to the 30-minute ceiling every time. A grep -c || echo 0 double-print anti-pattern that produced false WARNING messages downstream. Both were root-caused, fixed, and re-drilled to a clean end-to-end pass before the script was trusted with real operations. That work is exactly what a drill is for — finding bugs in the safety net before the safety net is what stands between you and being jailed.

One additional operational lesson surfaced during the drill: Horcrux's own systemctl ActiveState can report active before its Raft cluster has actually reformed quorum well enough to co-sign. Same category of "healthy-looking status ≠ real capability" issue as the chain's own voting_power field can produce, now confirmed to extend to the signing layer itself. The failback script now waits for genuine signed events, not for ActiveState=active, before declaring signing resumed.

The full topology and drill results live on the validator page. That page has the numbers; this piece has the shape.

// the first governance vote

Three weeks in, the first governance vote came up. Kava proposal 220 — recover the IBC light client for the Kava-Secret channel. Vote was YES.

The operational shape of casting a vote as a validator is different from casting one as a delegator. As a delegator, you click through a wallet UI. As a validator, you sign a governance transaction from the operator account — not the signing key, the operator key — using a discipline the validator page documents as "brief-import." The operator key is not resident on any always-on machine. It gets imported briefly for the vote, the vote transaction is signed and broadcast, and the key is removed again. Same principle as the signing-key discipline, applied to governance: the sensitive credential is never around when it doesn't need to be.

The substantive reasoning on the vote — why yes, what the recurrence of this fix pattern surfaces about relayer-set health on the Kava-Secret channel, why the meta-governance abstain vote is defensible even if it's not what I voted — lives on the on-site governance record. This piece is about the operational experience of casting the vote, not the merits of the proposal. That's a genuine split of concerns: the validator page and the governance record are the accountability artefacts, this piece is the story.

Cross-link to the governance record for the substance: /validators/kava/governance/220/.

// what would have been day-one doctrine

Some of the war stories in this piece became doctrine that got applied to the second chain. If I were starting Kava again from scratch, this is what would be day-one design rather than lesson-learned:

Startup ordering. Horcrux cosigners must be up and connected before the validator node starts its signing daemon. There's a roughly 3-second privval handshake timeout that will crash-loop the daemon if it opens its listener before a cosigner is ready to respond. Every operational script now has cosigner-side up before node-side up as a hard sequence.

The empty-placeholder-key gotcha. Some chain forks panic on startup if the priv_validator_key.json file is present but empty, rather than if it's absent entirely. The safe posture is to remove the file, not zero it. Deleted, not blanked.

AppHash verification before snapshot mount. The Polkachu-snapshot lesson from earlier. Verify the canonical AppHash from a public RPC before starting the long extraction. Fifteen seconds of check, potentially days of saved rework.

Cosigner and node physical separation as strict rule. Not "we should keep them separate where possible" — a hard rule that a cosigner share must never live on the same host as a signing node. Applied to Akash without exception. Would have been applied to Kava the same way if I'd had the discipline at day one.

Drilled failover, not planned failover. A failover script that has never been run against a real outage is theatre. HA4.5's discovery of two real bugs in a script that looked correct is the general case, not the exception. Drill the failover — with a real systemctl stop, not a simulation — before trusting it with a real incident.

None of these are theoretical. Every one of them was learned by doing the work slightly wrong the first time, and none of them cost real slashing because the safety architecture caught the mistakes before they compounded. That's what the safety architecture is for. But knowing what would have been day-one doctrine is the compounding gain — the second chain benefits from every one of these lessons without having to re-learn them.

// closing

Kava is chain #1. Twenty-eight days operational, 100% uptime, two delegators, one governance vote cast. The build wasn't dramatic. The delegator was.

The next chain is Akash. This time the shares distribution is day-one doctrine, not lessons learned later. The startup ordering is day-one doctrine, not a crash loop caught in staging. The snapshot AppHash check runs before the extraction, not after the third failed replay. The failover script gets drilled against a real outage before the validator ever bonds, not weeks after.

The homelab-first framing lands here: the validator playbook doesn't come from cloud infrastructure or a dedicated hosting provider. It comes from the rack, running alongside the rest of the workloads, doing the drilled operational work that most delegators trust cloud providers to handle. Whether that's the right posture for delegators is a conversation for someone else's article. Whether it's a defensible posture for an operator — one where the same person who wrote the failover script has run it, on real hardware, against a real outage — is what the whole series is trying to demonstrate.

Chain #1 signed its first block without ceremony. Chain #2 will sign its first block knowing what the first chain took a month to teach.

Two TrueNAS servers, one rack — the CORE-and-SCALE split I never designed

JJozzieTech — Thu, 23 Jul 2026 19:39:12 +0000

Two TrueNAS servers. Same rack. One's on CORE, one's on SCALE.

If someone asked me why, the honest answer isn't a design decision. It's a series of arrivals — one workload trigger at a time, each landing on whichever server had the shape to accept it, until the current arrangement crystallised into "how it is." Years of small choices, no whiteboard, no reorg since.

The segmentation piece covered a design I made deliberately. The OPNsense piece covered a build I saw through carefully. This piece covers a deployment that neither framing fits — because I didn't design it clean, and I didn't build it as one project. I ended up with it.

That distinction matters, because "designed," "built," and "ended up with" are three different things at homelab scale. Public writeups tend to blur them into one narrative — here's what I set out to do, here's what I ended up with, and the middle is a series of clean decisions that led logically from one to the other. Reality is messier. Workloads pivot. Constraints move. Servers get added because the workload you were adopting required them. Pools get shaped by an HBA ceiling you don't have anymore. Personal storage lands wherever there's room. And the rack ends up looking like the shape of a decade, not the shape of a design.

That's the story of my TrueNAS deployment. This piece walks through it — CORE first, SCALE later, the workload split as it stands, what actually shaped it, and the consolidation question I keep not answering.

// CORE came first

CORE went in as the storage layer for XCH — Chia — plots.

At the time, XCH needed one thing above all others: quantity of storage. Plot files are around 100 GB each and you want as many as your hardware can accept. That shaped every early decision about the box.

The R630 was the choice — a Dell 2U server, dual Xeon, plenty of ECC memory, and the important bit at the back: an external SAS HBA feeding a JBOD full of drives. The R630's own drive bays weren't enough; the JBOD was the whole point of the build.

The HBA was a SAS 6 Gbps card, and it had a drive-count ceiling I no longer remember precisely. What I remember is that it had one, and it was low enough to shape how pools got laid out. Rather than one large pool with vdevs stacked into it, I ended up with many smaller pools, each sized to what a single vdev could hold under the HBA's ceiling. That accretion of small pools is still visible in the dashboard today — vdev counts, disk-per-pool counts, all shaped by a constraint that stopped mattering when I sold most of the drives.

I ran a lot of 3 TB and 4 TB HDDs in that era. Most have since been sold as XCH scaled down. What CORE runs today is smaller than what it was built for: XCH still, and MMX — a Chia fork with its own harvester. The server is real hardware, the storage is real ZFS, and both are under-utilised relative to their original scope.

// SCALE came in with Storj

SCALE arrived when Storj replaced XCH as the direction of growth.

Storj is a distributed storage network — user data is chopped up, encrypted, and spread across independent nodes worldwide. My server hosts pieces of other people's data and serves them back on request. Different operational shape from XCH: instead of quantity of static plot files, it's active read/write traffic against smaller data volumes.

The relevant fact for this piece: Storj had a first-class TrueNAS app for SCALE, and none for CORE. Installing Storj natively meant SCALE. Running Storj on the R630 would have meant a manual container deployment, or a Docker VM alongside CORE, or something else that would carry more operational burden than clicking Install on the SCALE app catalog.

So I added SCALE. Not because I preferred it as a platform. Because the workload chose the platform.

SCALE went onto the Dell 740xd — a newer chassis than the R630, running SCALE as a VM on Proxmox on that machine. That let SCALE coexist with other workloads on the same physical server without turning it into a dedicated NAS. Storj lives there. MMX has some workload there too, alongside CORE's harvester. And over time, personal storage — media files, personal NAS shares — landed on SCALE because that's where the newer capacity was, and because SCALE's app catalog was better developed for the shape of workloads that came later.

CORE was chosen for XCH. SCALE was added for Storj. Everything else came in based on whichever server had space and shape at the time it needed to land.

// the split as it stands

Here's the current arrangement in the shortest form.

CORE (R630): XCH farming, MMX farming. Storage is JBOD-backed, structured as multiple small pools shaped by the historical HBA ceiling. The server has more capacity than the workloads currently need.

SCALE (740xd, as a Proxmox VM): Storj, some MMX, personal NAS, media files. Multiple pools, mixed drive shapes, one larger pool separating personal storage from DePIN workload storage.

No replication between the two servers. No cross-platform snapshot strategy today. Backups on both are on the horizon as part of a separate Proxmox Backup Server project — the natural place for that to land is one PBS instance handling both TrueNAS deployments plus everything else in the rack, rather than each TrueNAS doing its own thing.

The two servers don't talk to each other beyond being on the same network. They're not a cluster, they're not a pair. They're two independent deployments that happen to share a rack.

// what's shaped this beyond design

Four things shaped the current arrangement that weren't design choices.

The HBA drive-count ceiling. Real constraint in the early years. Made "many small pools" the shape rather than "one large pool with stacked vdevs." That shape persisted after the constraint stopped mattering — because reshaping ZFS pools is destructive, and there was never a moment worth spending the migration cost.

The XCH-to-alternatives pivot. When XCH scaled down as a workload, the drives went with it. What was left was capacity, not deliberate architecture. If XCH had stayed the primary workload, I probably wouldn't have added SCALE at all — the R630 alone was doing what needed to be done.

Storj's app requiring SCALE. This is the specific technical trigger. Storj's TrueNAS integration was on SCALE. If Storj had shipped a CORE plugin, or if I'd been willing to run Storj in a manually-managed container on CORE, SCALE might never have come in. One workload's platform choice became the reason for a second TrueNAS deployment.

Accretion of what came later. Personal storage, media files, the second-order workloads — they landed on SCALE not because SCALE was the right platform for them, but because SCALE was newer and had capacity. That's not architecture. That's opportunistic filling of empty space.

The pattern that connects all four: each specific arrival made sense on its own; the aggregate is a shape nobody would have designed from a blank sheet.

That's not a failure. It's what actually happens when you run infrastructure for years without a clean-slate migration event.

// the consolidation question

The natural next question: would I consolidate?

The criterion I'd apply is hardware footprint. If consolidation reduced the number of physical servers running TrueNAS in the rack, it would be worth doing. If it just moved workloads around without changing the physical footprint, it wouldn't.

That criterion doesn't currently trigger. Here's why.

The R630 is dedicated to CORE. Consolidating CORE's workloads onto SCALE would free the R630 — real hardware footprint reduction. But it would require migrating XCH and MMX plot data to SCALE, which is terabytes of data and a maintenance window I don't have. Both are non-trivial.

The 740xd runs SCALE as a VM alongside other workloads. Consolidating SCALE's workloads back onto CORE would free the SCALE VM but not any physical hardware — the 740xd would still be running other things. So the consolidation gain there is smaller.

Both servers currently run stable. No degradation pressure, no failing hardware, no capacity ceiling that forces the question. The forcing function isn't present.

PBS project is the natural precursor. Before consolidating anything, I want backup and snapshot posture solid across both platforms. That's the Proxmox Backup Server project sitting alongside this one on the roadmap. Consolidating without settled backup posture would be a much bigger risk than consolidating with it in place.

So the status is: not planned, not off the table. If the R630 develops a fault that would require investment to fix, that's the moment consolidation becomes real. Until then, both run.

The under-utilised capacity is what makes consolidation worth thinking about. The migration blast radius and the missing backup posture are what make it worth deferring.

Both those things are true simultaneously, and that's fine. Operator-honest posture doesn't require every decision to be resolved. Some decisions are open by design.

// closing

Two TrueNAS servers in one rack because history shaped it that way. CORE for XCH first, SCALE added because Storj required it, workloads accreted onto both from there.

The coming quarter doesn't change either of those facts. What changes is the PBS project settling backup and snapshot posture on both platforms — the piece that has to happen before consolidation is even a coherent question. After that, a hardware event, a workload consolidation, or a maintenance window I don't have yet.

Until then, both run.

If you're running a homelab TrueNAS today and wondering whether to migrate from CORE to SCALE — or the reverse — the answer isn't in a platform comparison. It's in your workload stack. Whether the apps you run natively on one but not the other justify the migration cost, and whether your backup posture is ready to make the migration safe. If either answer is no, the honest posture is to leave it where it is.

Sometimes the operator-grade decision is to leave things the way you found them and put the migration budget elsewhere. This is one of those decisions. Both TrueNAS servers run, and probably will for some time.

OPNsense on bare metal — 18 months as the only perimeter

JJozzieTech — Thu, 16 Jul 2026 20:32:03 +0000

For most of the homelab's life, there wasn't a firewall.

The ISP router did what ISP routers do — NAT out, block unsolicited inbound, don't ask hard questions. That's not zero security. It's just not intent-driven security. Which was fine, because the rack ran things I wrote for myself, and nobody on the internet was paying attention to my home connection.

Then I stood up an Akash provider.

The segmentation piece walked through what changed inside the rack when Akash landed — VLANs, ACLs, east-west policy. This piece is the parallel story of what changed at the edge. OPNsense on a dedicated box, ISP terminating on the WAN interface, all internal traffic on a single trunked LAN interface, running for 18 months as the only thing between the rack and the internet.

I'll cover the origin, the hardware, the split of labour between OPNsense and the Nexus 3548P, the four war stories that taught me the platform, and the four things I still haven't done that I know I should.

The last section is the point of the piece. Retrospectives that only list what worked are marketing. Retrospectives that admit what's still on the list are useful.

// no firewall, then one firewall

Before Akash, the perimeter was the ISP router.

Not zero security. Consumer-grade routers do block unsolicited inbound by default — that's been true of every ISP-supplied box shipped in the last twenty years. But the operative word is default. There was no policy. The perimeter existed because the ISP router existed. Anything more specific — logging, per-service inbound rules, outbound restriction, awareness of what was actually crossing — didn't exist because I hadn't asked for it.

That worked for a rack running things I wrote for me, running when I was watching. When Akash landed, the calculus changed. The Akash provider accepts inbound tenant deployments at an ingress that is by design publicly discoverable on-chain. Trusting the ISP router to be the only thing between that ingress and my workstation stopped being defensible once the provider was live and taking inbound deployments.

The natural sequence, once I was building for that: segment first, then police. The segmentation piece covered the segment step. This is the police step.

OPNsense went in on a dedicated box. ISP terminated on its WAN interface. Every internal VLAN — management, workload VLANs, storage VLAN — trunked into a single LAN interface. The rack now had two policy-driven layers separating it from the internet: the segmentation inside, and OPNsense at the edge. Neither trusts the other by default.

// why bare metal, not a VM

The obvious cheap answer would have been an OPNsense VM on Proxmox. It would have run fine. I didn't do that. Three reasons, in the order I actually thought about them.

Compute belongs to workloads. The Proxmox hosts exist to run DePIN infrastructure — Akash, storage, harvesters, validators. Every VM slot I give to a perimeter firewall is a slot I don't give to something earning. Bare metal at the edge means the servers stay full of what they're for.

The perimeter is a different animal from the compute. The firewall shouldn't share a fate with the hypervisor. If Proxmox goes sideways — kernel panic, storage-driver bug, upgrade gone wrong — the perimeter should be uninvolved. Bare metal enforces that separation physically, not just conceptually.

ISP terminates here. OPNsense is the box the ISP handoff talks to. That relationship is cleaner when exactly one physical device owns it, and when that device isn't a guest on a hypervisor whose primary job is something else entirely.

The hardware, as it exists in the rack today:

model:      dell optiplex 7080 SFF
cpu:        intel core i7-10700 (8c/16t)
memory:     32 gb
storage:    SATA SSD, built-in bay
nics:       1× onboard intel i219  (WAN)
            4× intel i350 add-in card  (LAN + 3 spares)
os:         opnsense 25.7 (freebsd base)
form:       SFF, rack shelf

The i7-10700 is overkill for what OPNsense currently asks of it. This rack doesn't push enough traffic to stress an 8-core Comet Lake at the edge. But headroom is what enables the still-on-the-list items — IDS at line rate needs cores, and a CARP secondary needs a matched-pair budget to be defensible.

Between OPNsense and pfSense: open-source posture and the UI. The plugin ecosystem sealed it.

// the split of labour

The segmentation piece established the split. Sharpening it for this piece:

The Nexus 3548P handles east-west. Inter-VLAN routing at line rate on the switching ASIC. Workload-to-workload isolation lives on the Nexus, enforced by ACLs on each SVI.

OPNsense handles north-south. WAN, NAT, port-forwards, outbound NAT posture, inbound perimeter policy. Anything that crosses the boundary between the rack and the internet.

Two rulesets, two devices, two audit surfaces. Both stay simpler.

Interface layout at the OPNsense end:

WAN interface   →  ISP handoff, single link, dhcp from provider
LAN interface   →  tagged trunk to nexus, carries all internal vlans
spare ports     →  available for CARP secondary or dedicated management

One physical LAN interface is enough because the workload VLANs are tagged on the trunk. OPNsense sees each VLAN as a virtual interface on top of the physical LAN port and can write rules against each independently — inbound and outbound, per-VLAN, without the physical port count constraining what's possible.

The spare ports on the add-in card matter for what's coming. When the CARP secondary goes in, pfsync between the pair needs its own link. That's what one of the spares becomes.

// what the rack does with NAT

OPNsense defaults to Automatic outbound NAT: one masquerade rule per LAN interface, everything else inferred. That's what most homelab OPNsense builds run. It's the correct default. It also isn't what this rack runs.

This rack runs Manual outbound NAT. Every workload VLAN has an explicit outbound rule referencing a named network alias. The pattern:

alias:  workload-a-net   →   10.x.x.0/24

rule:   source workload-a-net   →   NAT to WAN interface address

Two reasons that's worth the extra config surface.

Explicit is better than inferred. When you're staring at a firewall trying to work out whether workload X can reach the internet, seeing an explicit named rule beats reasoning about Automatic-mode inference every time. The rule either exists or it doesn't. There's no "well, in Automatic mode it would generate…" middle ground.

Aliases are reusable. The same workload-a-net alias that appears in the outbound NAT rule can be referenced in inbound rules, port-forwards, floating rules — anywhere the subnet needs to appear. One object, one place to update. When a workload's subnet changes — or when it moves — one alias edit propagates everywhere.

The trade-off: Manual mode means new workloads don't get NAT for free. Adding a workload means adding an alias, adding a rule, and applying. That friction is a feature. It means every workload on the wire had a moment where someone explicitly decided how it talks to the internet. Nothing gets to the outside by accident.

// four things OPNsense taught me

Four incidents from 18 months in. Each taught a different class of lesson.

1. The saved config that wasn't the applied config.

A static route to a workload subnet, correctly configured in OPNsense — and workloads on that subnet couldn't reach the internet. Firewall rules checked out. NAT checked out. The routing table on the L3 switch was correct. The route was in the OPNsense config screen but not in the OPNsense routing table.

Pending changes had been staged and never applied. Clicking Apply activated the route and connectivity came back immediately.

Lesson: saved is not applied. OPNsense stages changes to firewall, NAT, and routing configs, and requires an explicit Apply to activate them. The running config can silently drift from the staged one if you forget. Any troubleshooting session that starts with "but the config looks right" should also check whether pending changes are waiting.

2. The subnet OPNsense wouldn't accept as a NAT source.

Configuring outbound NAT for a workload VLAN. Enter the workload subnet as the source. OPNsense refuses: "not a valid source IP address or alias."

The subnet was valid. The rule editor just wouldn't accept it in raw form.

The fix: create a named network alias containing the subnet, then reference the alias in the rule. Alias workload-a-net containing 10.x.x.0/24, and the rule takes it without complaint.

Lesson: OPNsense treats aliases as first-class objects. Some rule fields insist on aliases even when a raw CIDR is technically valid. This isn't a bug — it's the OPNsense-idiom answer to "how do we make firewall rules maintainable at scale." Once the reflex to reach for aliases is in place, everything gets more consistent.

3. The firewall rule that wouldn't edit.

An inbound WAN rule showed up in the firewall ruleset but the edit UI wouldn't let me change it. It looked like a normal rule. It just wouldn't respond to edits.

The rule was auto-generated by a NAT port-forward. OPNsense's port-forwards have a Filter rule association setting that automatically creates the corresponding firewall rule and manages it as a dependency of the port-forward. Direct edits either don't apply or get overwritten the next time the port-forward is touched.

Fix: either edit the port-forward's own rule fields (which flow through to the generated firewall rule), or set the association to "None" and manage the firewall rule separately.

Lesson: know which rules OPNsense controls and which you control. Not every rule in the ruleset is an independent object. Assuming otherwise costs you an afternoon.

4. The package update that wouldn't complete.

A routine package and firmware update ran into a repository / package-consistency issue and stopped mid-upgrade. Recovering the box required diagnosing the state of the pending upgrade rather than just retrying. Endpoint landed at OPNsense 25.7.3_7 eventually.

The firewall stayed on-line during the diagnosis. That was luck. I hadn't taken a config backup before the update, and I hadn't confirmed console access. If the update had left the box in an unusable state, I would have been rebuilding OPNsense from memory.

Lesson: before touching updates, capture a config backup and confirm console access. Both take five minutes. Neither had happened. Both are habits now.

// what I still haven't done

Four gaps. In the order I'd actually tackle them if I started this weekend.

1. Backups first.

The single most embarrassing gap. OPNsense's config is an XML export that can be committed to Git, uploaded to cloud storage, or scheduled to a network share via the built-in backup module. None of that has happened here.

Backups are first not because they're the most urgent to have, but because they're the cheapest to get. Ten minutes of config setup and I have a restorable state. Every subsequent gap-closing effort is safer once this is in place. The next OPNsense update — and it'll happen — will be less nerve-wracking with a known-good config sitting in a repo somewhere.

2. HA second.

Would I make the same bare-metal call again? Absolutely. Would I have planned CARP into the initial build from day one? Also absolutely.

Retrofitting HA to a running perimeter is harder than designing it in. CARP virtual IPs need to be planned around the WAN handoff. pfsync between peers needs a dedicated link. Both are easier to reason about with a matched pair of boxes than with an active production box and a second box being brought into a pair. The next OptiPlex is the plan. It's a project, not a config change.

3. Logging third.

OPNsense's logs currently live on the box and roll over. That's fine for "check the last hour's blocked traffic." It isn't fine for anything else. The natural target is either a Loki/Grafana stack in the rack or Graylog on a dedicated VM.

Logging goes before IDS/IPS because IDS/IPS generates the logs. Having nowhere for them to land defeats the point.

4. IDS/IPS last.

Suricata on the LAN interface, ET Open ruleset, IDS mode first (observation) before IPS mode (enforcement). Deliberately last, because the false-positive load is real, the tuning effort is real, and the payoff without steps 1–3 is limited. IDS/IPS without a config backup, without HA, and without a place to send logs would just add fragility instead of catching it.

The order is deliberate. The temptation with lists like this is to lead with the exciting-sounding gap. Backups aren't exciting. They're the load-bearing gap that everything else depends on.

// closing

The perimeter has held for 18 months. Zero incidents I've traced to the firewall itself. Two of the four war stories above (the un-applied config, the un-editable rule) taught me how the UI thinks. Two (the alias requirement, the package update) taught me how the platform thinks. All four made me more useful, not less.

The next 18 months aren't going to be about new capability. They're going to be about closing the four gaps. A backup piece. A CARP-HA piece. A logging-to-something-real piece. Eventually an IDS/IPS piece, once the first three are in place to catch what it finds.

If you're running a homelab without a firewall today because the ISP router is fine, you're where I was two years ago. The trigger to move isn't a security incident — those don't announce themselves. The trigger is any workload you'd rather not have sharing a box with your workstation, or any inbound traffic you can't defend by hand-waving at NAT.

If you already have OPNsense running: check whether your config is backed up. That's the single highest-leverage thing you can do this week.

How I got a suspended TrueNAS pool back online without replacing a single disk

JJozzieTech — Thu, 09 Jul 2026 02:54:20 +0000

The rack had just come back online after an outage. Most things looked fine — VMs booted, the network reconverged, monitoring caught up on the missed intervals. Then I ran zpool status on one of the TrueNAS SCALE nodes and saw output I hadn't seen in production before.

An 11-disk RAIDZ1 vdev. Multiple drives marked as faulted. One member listed as removed. The pool itself in SUSPENDED state.

The reflex, in that moment, is to start replacing disks. That reflex is wrong — at least as the first move. This piece is how I got the pool back online without touching a single drive, and what I learned about how ZFS reports failure states along the way.

The pool in question is on my TrueNAS SCALE machine, which runs alongside a separate CORE box on a different chassis. The recovery pattern is the same on either — this isn't SCALE-specific ZFS behaviour — but the incident happened on SCALE, so that's where the console output in this piece comes from. For privacy, I'll refer to the affected pool as MediaPool-A.

// what the console showed

The output was the classic suspended-pool warning:

  state: SUSPENDED
 status: One or more devices are faulted in response to IO failures.
 action: Make sure the affected devices are connected, then run 'zpool clear'.

Then the vdev listing. The RAIDZ1 topology looked much worse than it turned out to be — most of the 11 disks were showing as faulted, and one was marked as removed. If you'd shown me that output out of context and asked me to guess what happened, I'd have said "shelf died" or "backplane failure." The vdev looked shot.

But there was a shape to the failure that mattered. Multiple disks going down at the same time is exactly what you'd expect if something they shared had gone down — an HBA path, a SAS cable, a backplane, a controller — rather than the disks themselves failing individually. Simultaneous multi-disk failure is one of those things that sounds catastrophic and usually isn't. Independent failures don't cluster in time. Shared-path failures do.

That's the diagnostic reframe the piece is built around. Before you touch a disk, ask what those disks have in common that isn't the disks.

// the reflex to resist

The most important part of ZFS recovery is what you don't do in the first ten minutes. Here's the list, in order of danger:

Don't add disks to the pool. Whatever's shown as unassigned in the TrueNAS GUI may not actually be unused. If the pool is suspended and the disks are still recognised by the OS, they're still valid pool members even if the GUI is being coy about it. Adding them to a new vdev structure is the fastest way to permanently lose the pool.
Don't wipe any disk. Same reasoning. If a disk still has valid ZFS labels for this pool, wiping it is destructive.
Don't replace any disk. Not until you've confirmed there's a disk that actually needs replacing. See earlier point on multi-disk-fault patterns.
Don't export or destroy the pool. zpool destroy is exactly what it sounds like. zpool export is safer but shouldn't be done while the pool is in an unstable state.
Don't start a scrub while the pool is suspended. A scrub is not the recovery mechanism when the pool has been suspended due to I/O access problems. The pool needs to be brought back into a stable online state first. Scrubbing before that either does nothing useful or, worse, hammers the disks while they're on an unreliable path.

That last one catches people. Scrubs are the go-to "is everything OK?" tool in ZFS-land, and the muscle memory when something looks wrong is to run one. Muscle memory is wrong here. A scrub is validation, not recovery. It belongs at the end of the recovery process, not the start.

// step one: are the disks actually there?

Before running anything against the pool, I checked whether the operating system could see the disks. This is the cheapest possible diagnostic and it usually tells you which world you're in.

smartctl --scan

The disks were visible. All of them. That was the first good sign — it meant the OS could see the hardware, which meant whatever had happened during the outage had resolved at the hardware/path level, which meant the pool was recoverable in software without touching any physical component.

Then I checked kernel logs for signs of ongoing HBA or SAS trouble:

dmesg -T | grep -iE "sas|scsi|mpt|mps|mpr|reset|timeout|failed|medium error|sense|I/O error|device offline|link"

The kernel log showed events from the outage itself — expected, since the disks had briefly gone away and come back — but nothing recent. No active storms of SAS resets. No timeouts still firing. No disks going offline. The path was stable now, whatever had happened during the outage.

That was the second good sign, and it was the one that gave me confidence to move to the recovery step. If the kernel had still been throwing SAS resets or timeouts in the current second, I would have stopped, escalated to the physical layer — cables, HBA, PSU — and only come back to ZFS after the path was clean. Running zpool clear against a still-unstable path just re-triggers the fault, and does it at higher urgency this time.

// zpool clear, safely

Once the two "is the path stable?" checks came back clean, the recovery command was the one ZFS itself had already suggested:

zpool clear MediaPool-A
zpool status -v MediaPool-A

The pool returned to an ONLINE state. One disk briefly showed a small number of checksum errors on the first status check after the clear — expected, because the outage had generated some genuinely bad reads that ZFS had queued up. A second zpool clear zeroed the counters, and the follow-up status came back clean:

MediaPool-A = ONLINE
all disks = ONLINE
READ / WRITE / CKSUM = 0
errors = No known data errors

No files reported as corrupted. No permanent errors. The pool was back.

The recovery itself was two commands. The rest of the work was in the "is the path actually stable?" checks before them — cheap to run, and the difference between a clean clear and re-triggering the fault at a moment when the pool was already suspended.

// what actually happened

The most likely cause was a temporary I/O path disruption during the outage. Something in the shared path — HBA, cable, expander, backplane, or the disks all briefly losing power at slightly different times as the rack came back — presented as multiple simultaneous device failures to ZFS. ZFS did exactly what it's designed to do: suspended the pool to protect the data from being written to devices it wasn't sure were healthy.

Once the path stabilised, the disks were visible again. zpool clear told ZFS "the fault condition is resolved, retry," and ZFS did. Because the disks and their data were still intact underneath, the retry succeeded, and the pool came back with no permanent state loss.

The three ZFS terms that mattered here are worth stating precisely, because their intuitive meanings are misleading:

FAULTED does not always mean a dead disk. It means ZFS has decided this device is unreliable enough to stop trusting for the moment, based on the errors it's seen. A device can be FAULTED for reasons that have nothing to do with the disk itself.
REMOVED does not always mean physically removed forever. It means the device that was here at some point isn't visible right now. It can be transient.
SUSPENDED does not always mean the pool is lost. It means ZFS has stopped I/O on the pool to protect it, pending resolution of whatever fault condition triggered the suspension.

Context is what tells you which of those interpretations applies. Multiple simultaneous FAULTED devices, immediately after a rack outage, on drives you can still see with smartctl and don't have active kernel errors on, points at "transient path disruption during the outage" and away from "disks died." A single FAULTED device weeks after normal operation points the other way, and the response is completely different.

// what I'd make instinctive

Two things, and one thing I already had right.

Look at the shared layer first. When multiple disks fault at once, the useful question is not "which disks failed?" but "what do they share that could have failed?" HBA, SAS path, backplane, PSU, controller firmware. Independent failures don't cluster in time; shared-path failures do. Every panic-response-to-multi-disk-fault I've ever done has, eventually, come back to something at the shared layer. Skipping straight to the shared layer as the first diagnostic saves you the panic.

zpool clear is safe only after the path is confirmed stable. It is not a magic fix. If the underlying path is still unstable, zpool clear re-triggers the fault immediately, and the second suspension arrives faster than the first because ZFS has now seen the pattern twice. Confirming path stability first — smartctl --scan, then dmesg | grep for active errors — takes about thirty seconds and is what makes zpool clear a recovery tool instead of a way to make things worse.

The thing I had right: not scrubbing while the pool was suspended. That's an instinct worth keeping. Scrub validates a stable pool. It doesn't recover an unstable one.

// closing

This was a clean recovery because the underlying condition turned out to be transient. The pool came back with no touch to any physical disk, no wipe of any data, no rebuild of any vdev. The recovery cost was two commands and about ten minutes.

Not every ZFS recovery is that clean. The same outage that suspended MediaPool-A also took down a second pool on the same box in a different, harder way — one where an entire RAIDZ1 vdev appeared to be gone, and the recovery required reading ZFS labels directly to prove the disks were still there before running any of the commands above. That's the next piece.

If you're recovering from something like this now: check the disks are visible, check the kernel logs are quiet, then run zpool clear. In that order. Everything else you might be tempted to do can wait.

I run my homelab like a miniature data centre — here's the network design that made it possible

JJozzieTech — Thu, 02 Jul 2026 06:57:36 +0000

The homelab started flat. One /24, everything on it. My workstation, the NAS, the Proxmox host, and — over time — a growing list of workloads sharing the same broadcast domain because that was the path of least resistance.

For a while, that was fine. A homelab running one workload doesn't need segmentation any more than a house needs an office door.

Then I stood up an Akash provider.

An Akash provider is, in shape, a Kubernetes cluster that accepts inbound tenant workloads from the internet — real deployments, paying for compute, containers I didn't write landing in namespaces on my hardware. The provider itself is documented at github.com/jjozzietech/akash-provider-ops-public — this piece is about the network underneath it. The containerisation posture itself is fine. I trust the isolation model. But trust isn't a network design. And the network at that moment had the tenant workload cluster sitting on the same subnet as my workstation, my NAS, and my Proxmox management interface.

That was the moment I stopped thinking of the rack as a home network with extra boxes, and started thinking of it as a small data centre.

This piece is the network design that came out of that shift. I'll cover the layout, the rules that hold it together, and the Nexus and Proxmox configs that anchor it — with the specifics of my own deployment sanitised. It's not a step-by-step replication guide. It's the design pattern, with enough of the shape to be useful and enough restraint to not double as a recon document for my own rack.

// the original design

The flat layout looked like this:
home lan — 192.168.1.0/24

opnsense (perimeter)

cisco nexus (dumb L2 switching)

proxmox host

workload VMs (all on the same subnet)

What it got right: zero routing complexity, everything reachable from everywhere, fast to stand up. If you're running one project on a homelab, this is the correct design. Don't over-engineer it.

What stopped working, as soon as the second project landed on the rack, was that the design had no answer to three questions:

How do I write different firewall policies for different workloads when they all share a subnet?
How do I keep a compromise in workload A from reaching workload B?
How do I troubleshoot when every packet in the rack lives in the same broadcast domain?

The last one is the operational tell. When "which workload is chatty on the wire right now?" becomes a question you can't answer from a switchport counter, the design has outgrown the problem it was designed for.

The industry term for what I was worried about is east-west traffic — traffic between workloads on the same tier, as opposed to north-south traffic between workloads and the outside world. Flat L2 doesn't give you any policy handle on east-west. You just have to trust that nothing on the wire behaves badly.

That's a defensible assumption for a house LAN. It's not a defensible assumption for infrastructure that hosts tenant workloads.

// the reframe: stop thinking residential

Commercial data centres don't put every workload in the same subnet. They isolate services into dedicated network segments, each with its own routing policy, its own firewall posture, its own blast radius. When something goes wrong, the fault is bounded. When something needs a different security posture, it gets one. When a workload needs to move to different hardware or a different site, the network context comes with it.

That's the model I took. Every project on the rack gets:

Its own VLAN
Its own /24
Its own firewall policy at the routing boundary
No default trust with any other VLAN

The hypervisor stays on management. Workloads live on their own VLANs. Ops access is explicit, not implicit.

A quick note on hardware, because it matters. The switch doing this work is a Cisco Nexus 3548P — enterprise-grade L3 switching, ACLs applied inbound on the SVI, inter-VLAN routing at line rate on the switching ASIC rather than punted to a CPU. A UniFi or Mikrotik can do most of what's below, and if you're building this pattern on consumer gear, that's a reasonable place to start. What you give up is the ability to enforce policy at wire speed with the ACL evaluated in hardware. On a homelab-sized rack it's rarely the bottleneck. On a rack that hosts real workloads with real tenants, I wanted the ceiling raised.

// the architecture

Here's the layout, in the same YAML metadata style I use across the site:
management: 192.168.1.0/24 workstation, opnsense, proxmox host

vlan N (workload-a): 10.N.N.0/24 first project

vlan M (workload-b): 10.M.M.0/24 second project

vlan …: … future

Actual VLAN IDs and subnets in my deployment differ. The pattern is what matters, and I'll get to why the pattern is the way it is in the "what I'd do differently" section.

Three design rules underpin this.

The hypervisor stays on management, VMs live on workload VLANs. The Proxmox host is not a workload. It's the platform workloads run on. Putting it on the management VLAN means the host stays reachable when a workload VLAN goes sideways — backups continue, console access works, snapshots run. If the Proxmox host is on the same VLAN as a workload and that VLAN's routing breaks, you've just lost your recovery path.

One /24 per workload, gateway on the Nexus. The L3 boundary between VLANs is where policy lives. That means the Nexus SVI, not OPNsense. OPNsense's job is different, and I'll get to it.

Workload VLANs are peers, and no VLAN trusts another by default. East-west is denied. ACLs grant the exceptions. If workload A ever needs to talk to workload B, that's a deliberate rule, not a side effect of shared broadcast domain.

// what the nexus does

Three sanitised snippets, each with a sentence on intent.

VLAN definition:
vlan 10

name workload-a

Nothing exciting. The VLAN is a container; the interesting policy lives on the SVI.

SVI with gateway and inbound ACL:
interface Vlan10

description workload-a-gw

ip address 10.10.10.1/24

ip access-group acl-workload-a in

no shutdown

The ip access-group acl-workload-a in line is the important one. Inbound on the SVI means traffic entering the Nexus from the workload VLAN is what gets filtered. Any packet leaving workload A's subnet passes through that ACL before it gets routed anywhere else. That's how east-west enforcement happens at the routing boundary.

Trunk to a Proxmox node:
interface Ethernet1/3

description pve01-trunk

switchport mode trunk

switchport trunk allowed vlan 10,20,30

spanning-tree port type edge trunk

no shutdown

The switchport trunk allowed vlan list is explicit. Only VLANs I've deliberately added to the trunk can traverse it. New workload comes online → the VLAN gets added to the allow-list. Mundane, but it means the trunk isn't a wide-open pipe by default.

I leave the native VLAN unset on workload trunks. Tagging is explicit on the Proxmox side, which I'll cover shortly.

// ACLs are the actual security mechanism

VLAN segmentation isolates broadcast domains. That's it. Two VLANs on the same switch, both trunked to the same Proxmox host, can absolutely talk to each other — as soon as one of them has a route to the other. The route is provided by the SVI. Which is why the SVI is where policy has to live.

The pattern, not the rules, for a workload VLAN's ACL looks like this:
acl-workload-a:

permit management-workstation -> workload-a (ops access in)

permit workload-a -> dns, ntp (outbound essentials)

deny workload-a -> 10.0.0.0/8 (block east-west)

deny workload-a -> 172.16.0.0/12 (block east-west)

deny workload-a -> 192.168.0.0/16 (block east-west)

permit workload-a -> any (internet last)

Walking through the logic:

Rule order matters. The three RFC1918 denies sit before the permit-any-internet. Reverse them and the denies never evaluate — the permit-any matches first and every internal network is reachable. This is the single most common way I've seen this pattern implemented incorrectly.

The management workstation gets an explicit permit at the top. Without it, I lose ops access to the workload VLAN — SSH, monitoring, any control-plane traffic. With it, only that one host can reach in. Everything else that tries to reach workload A from a private range hits the RFC1918 denies further down.

DNS and NTP get their own permit lines. Cleaner than carving exceptions into the deny block. The workload needs to resolve names and know what time it is; both are explicit.

No VLAN-to-VLAN permits exist. If workload A needs to talk to workload B in the future, that'll be a deliberate exception — added, logged, reviewed. Default deny is the actual default, not aspirational.

What's not in this article: the per-host permits, the port restrictions inside the workload VLAN, the exact management-range definition, and the north-south inbound permits handled at the OPNsense boundary. The shape is the point. The full ruleset stays on the rack.

// dividing the labour: nexus vs opnsense

There's a clean split between the two devices.

The Nexus handles east-west: inter-VLAN routing and policy. Everything that stays inside the rack.

OPNsense handles north-south: WAN, NAT, port forwards, outbound NAT, anything that crosses the perimeter to or from the internet.

Two devices, two audit surfaces, two separate rulesets. Both stay simpler as a result.

The alternative — collapsing everything into OPNsense as a router-on-a-stick — is a design I see in a lot of homelab writeups, and it works. But it has two costs. First, inter-VLAN traffic becomes CPU-bound on the firewall instead of running at line rate on the switch. Second, north-south and east-west rules end up in the same ruleset, which makes both harder to reason about over time. You end up asking "does this rule apply to WAN traffic or inter-VLAN?" every time you touch the config, and the answer depends on interface assignments buried elsewhere.

Keeping the two roles physically separate — Nexus for east-west, OPNsense for north-south — is more disciplined. And on the Nexus, inter-VLAN routing doesn't touch the firewall at all, which means an OPNsense reboot doesn't interrupt workload-to-workload traffic. (It also doesn't enable workload-to-workload traffic, because the Nexus ACL denies it by default. But the point is that the firewall isn't in the east-west path either way.)

// the proxmox side

The Proxmox network config on the host is deliberately simple. One VLAN-aware Linux bridge, one LACP bond feeding it, management IP on the untagged side.

Here's the shape, sanitised — real interface names, redacted IPs:
auto lo

iface lo inet loopback
iface ens7f0 inet manual

iface ens7f1 inet manual
auto bond0

iface bond0 inet manual

bond-slaves ens7f0 ens7f1

bond-miimon 100

bond-mode 802.3ad

bond-xmit-hash-policy layer2+3
auto vmbr0

iface vmbr0 inet static

address 192.168.1.10/24

gateway 192.168.1.1

bridge-ports bond0

bridge-stp off

bridge-fd 0

bridge-vlan-aware yes

bridge-vids 2-4094

Two physical NICs (ens7f0 and ens7f1) bond together into bond0 using LACP (802.3ad) — the switch side is configured as a matching port-channel. That gives me two paths to the Nexus, active/active, with automatic failover if one link drops.

The bond feeds a single Linux bridge, vmbr0, configured as VLAN-aware. The bridge trunks every VLAN from 2 to 4094 (in practice, only the VLANs actually on the switch's allow-list traverse it). The hypervisor's management IP sits on the bridge itself on the untagged VLAN — which is the management VLAN, reachable from the workstation and the rest of the management infrastructure.

The VM side is one line of config per VM: the NIC gets a VLAN tag set in the Proxmox GUI. The VM sees a regular Ethernet interface. The bridge handles tagging on the way out to the bond, and the Nexus routes based on the tag. That's the whole story on the host.

Notably absent: a separate bridge per VLAN. That design works, but it doesn't scale — every new VLAN means a new bridge, a new set of uplinks to manage, and a growing surface of things that can drift out of sync between hosts. One VLAN-aware bridge trunked to the switch does the same job with dramatically less state to maintain.

// what I'd do differently

Two things: one process, one design. Plus the honest note on the migration itself.

Process: design the ACLs before the workload goes live, not after.

The segmentation itself I got right the first time. The ACLs I got right eventually — but I built them reactively, as workloads landed and I discovered gaps. New workload goes live → I stand up the VLAN and SVI → I permit the ops access I need to configure it → the workload comes up → I notice traffic leaking somewhere it shouldn't → I tighten the ACL. Each iteration got closer to correct. But each iteration also had a window where the ACL was more permissive than I wanted, and one of those windows was long enough to matter.

The right approach is to have a template for both directions of policy before you provision anything. The east-west deny block as a ready-to-paste chunk. The north-south permit list — which ports OPNsense will forward inbound, from where — drafted alongside the workload's own deployment plan. Apply both at the moment the VLAN is created. The workload never exists in a permissive state, even briefly. The "did I remember to tighten that rule?" question never comes up.

This is the single thing that separates the segmentation I have now from segmentation I'd have designed on day one if I could start over.

Design: I'd keep the VLAN scheme exactly as it is.

The pattern is VLAN N maps to 10.N.N.0/24. VLAN 10 lives in 10.10.10.0/24. VLAN 11 lives in 10.11.11.0/24. And so on.

It's opinionated and I'll defend it. The payoff is that troubleshooting is self-documenting. A packet capture shows a host in 10.10.10.0/24 and I know without looking that it's on VLAN 10. An ACL entry references 10.11.11.0/24 and I know without looking it's targeting VLAN 11. Every debugging session, every ACL review, every firewall rule audit is faster because the VLAN ID and the subnet are the same number.

The downside is you're locked into a /24 per VLAN, and you can't ladder easily beyond VLAN ~250 before you start running out of clean second-octet space. For a homelab, that's not a real constraint. If it ever becomes one, the failure mode is switching to a different scheme for the overflow VLANs — the existing ones don't need to change.

Migration: rolling, and it was fine — mostly.

The migration itself was rolling rather than greenfield. I had running workloads, no maintenance window, and no ability to stand up a parallel network to cut over to. So each project moved one at a time: stand up the new VLAN, ACL, and SVI on the Nexus; add the VLAN to the Proxmox trunk allow-list; retag the VM's NIC; update any OPNsense port forwards to the new IP; verify; move on to the next.

Here's the one that stung. My Storj storage node went dark for longer than I'd like to admit during the TrueNAS cutover. TrueNAS moved from the flat network to its own VLAN — new subnet, new IP. The Storj app runs as a TrueNAS CORE plugin, so it inherited the new IP automatically. What didn't inherit anything automatically was the OPNsense port-forward, which cheerfully continued sending inbound Storj traffic to TrueNAS's old address. From my side, everything looked fine — TrueNAS was up, the plugin was running, SMB shares were serving internal clients without complaint. From the Storj network's side, my node had stopped accepting inbound connections. It took a reputation dip before I noticed.

The failure mode is generic and worth naming: when a workload moves subnets, every downstream reference to its old address is now stale. Port-forwards, reverse proxies, DNS records, anything with a hardcoded IP in a config, anything scraping Prometheus on the old address. My cutover checklist was fine for the workload itself. It was not fine for everything that talked to the workload. That's the kind of gap a pre-provisioned north-south permit template — with every port and destination mapped out before the move — closes automatically. My checklist now includes an explicit "audit every reference to the old IP" step. It hasn't happened again.

// scaling: why this pays back

The marginal cost of adding workload N+1 is now:

Pick the next VLAN ID and its matching /24.
Define the VLAN and SVI on the Nexus, with the template ACL applied.
Add the VLAN to the Proxmox trunk allow-list.
Tag the VM's NIC.
Configure north-south forwarding on OPNsense if the workload takes inbound traffic.

That's a fifteen-minute job for a workload that doesn't need custom firewall policy, and maybe an hour if it does. Not a redesign. Not a rework of anything already running.

The same logical layout transfers to colocation or a commercial data centre with the physical topology changing but nothing else. The VLAN IDs, the subnets, the ACLs, and the OPNsense rules all move as-is. That was one of the original design constraints — build a homelab network that doesn't need to be rethought when the rack does — and it's the part I'm most quietly proud of.

// closing

Most homelabs don't need any of this. If you're running a Plex box and a media NAS and maybe one game server, flat L2 is the right answer. The complexity here only starts paying back when you're running multiple workloads with materially different trust postures — something you built yourself next to something you didn't, something that accepts inbound traffic from the internet next to your workstation, something in production next to something in perpetual dev.

The moment you cross that line, segmentation stops being over-engineering and starts being basic operational hygiene.

This design pattern is the substrate the rest of the rack runs on. The next few pieces in this series build on it — OPNsense as the perimeter, TrueNAS as the storage layer, and eventually the workload-specific pieces where individual services get their own writeups. All of it assumes the network underneath looks like this.

If you're running mixed workloads on a flat network today, the question isn't whether to segment. It's how many workloads you want to add before you do.

DEV Community: JJozzieTech

The wildcard cert saga — eight days from expiry, three failed attempts, one flag

// the first hour: what killed the renewal cron

// combining fixes: credential rotation plus architecture change

// first failed attempt: cert-manager's zone-walking bug

// second failed attempt: the retry loop became the story

// the pivot that wasn't really a pivot

// third obstacle: the --challenge-alias discoverability gap

// the turning point

// the fix that landed

// NGINX Gateway Fabric added one thing for free

// what I'd make instinctive

// closing

The error pointed at the GPU. The culprit was the HBA. — a Proxmox passthrough recovery

// what the GPU was actually there to do

// the diagnostic that didn't converge

// FLR: what makes an IOMMU group problem actually a passthrough blocker

// moving the HBA to a slot that mattered

// the second time was clean

// what I'd make instinctive

// closing

Recovering a ZFS pool with a missing RAIDZ1 vdev — without wiping a single disk

// what the console showed

// the critical move: don't trust the GUI

// zpool clear, this time with confirmation

// the second problem: permanent errors

// what SMART said afterward

// what I'd make instinctive

// closing

After a TrueNAS ISO reinstall — the two things that broke that no one warned me about

// why the fresh ISO was the answer

// the two things that break after reinstall

// the fix sequence

// what I'd make instinctive

// closing

A Kava validator on my existing stack — how the plan came together, and what the second chain will do differently

// the hardware happened to have the right shape

// the snapshot that wouldn't replay

// Horcrux 2-of-3: the signing architecture

// the last check before bonding

// first signed block, first delegator

// building the safety net: standby + drilled failover

// the first governance vote

// what would have been day-one doctrine

// closing

Two TrueNAS servers, one rack — the CORE-and-SCALE split I never designed

// CORE came first

// SCALE came in with Storj

// the split as it stands

// what's shaped this beyond design

// the consolidation question

// closing

OPNsense on bare metal — 18 months as the only perimeter

// no firewall, then one firewall

// why bare metal, not a VM

// the split of labour

// what the rack does with NAT

// four things OPNsense taught me

// what I still haven't done

// closing

How I got a suspended TrueNAS pool back online without replacing a single disk

// what the console showed

// the reflex to resist

// step one: are the disks actually there?

// zpool clear, safely

// what actually happened

// what I'd make instinctive

// closing

I run my homelab like a miniature data centre — here's the network design that made it possible

// the original design

// the reframe: stop thinking residential

// the architecture

// what the nexus does

// ACLs are the actual security mechanism

// dividing the labour: nexus vs opnsense

// the proxmox side

// what I'd do differently

// scaling: why this pays back

// closing

// third obstacle: the `--challenge-alias` discoverability gap