DEV Community: John Jacob (JJozzieTech)

Why I tested from inside the network

John Jacob (JJozzieTech) — Thu, 05 Nov 2026 19:59:25 +0000

The Internet upgrade was one of the final steps of my rack and network migration. An Aussie Broadband 2000/500 Mbps service had to work through the infrastructure actually carrying production traffic — not merely show a multigigabit link at the firewall. The path included a virtual machine, a virtualisation bridge, aggregated host uplinks, a redundant switching fabric, segmented routing, a firewall and the broadband handoff.

The question was not whether Aussie Broadband had sold me a 2 Gbps tier. It was whether a workload inside the rack could reach that tier, and what the measurements did — and did not — prove.

// what a speed claim actually measures

The contracted 2000/500 Mbps tier, the negotiated Ethernet interface rate, and application-level speed-test throughput describe different things. A 2.5GbE WAN interface can provide sufficient physical link capacity without guaranteeing an application 2 Gbps. Equally, a single lower test result does not establish an ISP shortfall.

My engineering report records that the upgraded service was active on 15 October 2026. A 24 October firewall interface capture shows the broadband-facing interface active at 2500Base-T full duplex. The physical media mode exceeds the nominal downstream service rate. This is necessary capacity, not a throughput measurement by itself.

// the path I measured

I ran the final benchmarks from a production service VM on a virtualisation host, rather than from the firewall console. Host networking used an IEEE 802.3ad bond built from two 10GbE links, traversing the switching and routed security edge to a 2.5GbE WAN. According to the source report, no 1GbE interface was present on the measured path. The test was conducted while the production compute platform was active; no service impact was recorded.

Figure 1. Generic logical path for the production-VM throughput tests. It deliberately omits product names, addressing and VLAN identifiers. This is a forwarding-path representation, not a diagram of active/standby member selection or a failover validation.

// nine runs across three networks

On 19 October 2026 between 06:03 and 06:06 AEDT, I ran Ookla Speedtest CLI 1.2.0 from that VM. Three Sydney networks — Telstra, Superloop and Optus — were tested three times each. The tests were sequential, interleaved across endpoints, and separated by approximately five seconds. Ookla's default multi-connection method was used, with JSON results retained for later review.

A tenth test to Aussie Broadband's own Sydney server was deliberately excluded from the nine-run off-net statistics, avoiding reliance on a single on-net measurement.

Metric	Nine off-net runs	What it means
Download median	2,069 Mbps	Measured from the production VM
Upload median	535 Mbps	Measured from the production VM
Idle ping median	2.57 ms	Idle latency, not loaded latency
Download range	2,013–2,077 Mbps	All nine runs exceeded 2,000 Mbps
Upload range	533–542 Mbps	All nine runs exceeded 500 Mbps

Both bandwidth medians were calculated independently from the nine off-net runs. The recorded throughput was consistent with delivery of the advertised tier under those test conditions. These measurements do not establish sustained all-day throughput or performance to every Internet destination.

// the earlier 936 Mbps result

An earlier firewall-local iperf3 reverse-direction test returned 936 Mbps at the receiver; its corresponding upload receiver figure was 455 Mbps. That client was the firewall itself, testing against a single remote server. This is a materially different test from a multi-connection speed test initiated on a production VM and repeated across three external networks.

The earlier result resembles a throughput ceiling often encountered around a 1GbE segment, but I have no proof that a 1GbE link caused it. Remote-server limits, stream behaviour and other test-path differences remain possible. In particular, the available evidence does not identify the limiting hop. I retain the 936 Mbps result as a useful diagnostic lesson, not as a resolved fault.

// latency belongs next to throughput

Bandwidth results were strong across all three test networks, but loaded latency was not uniform. The three Superloop download tests recorded 50.2, 55.2 and 44.4 ms of loaded latency, despite idle ping near 2.5 ms. The Telstra and Optus download-loaded values remained below 5 ms in these runs.

Figure 2. Download-loaded latency, three sequential runs per ISP (Ookla IQM metric). An endpoint-specific observation, not proof of the location or cause of congestion.

The reported packet-loss field was 0% for Telstra and Superloop runs. Optus did not report a loss value; it would be inaccurate to claim 0% loss for all nine runs. Likewise, the latency pattern cannot be attributed conclusively to queueing, shaping, peering or equipment without further path-specific measurements.

// what the result proves — and what it does not

The production traffic path delivered more than 2 Gbps downstream and 500 Mbps upstream to the selected off-net test endpoints, with a WAN physical negotiation of 2.5GbE. I could validate that result without removing the workloads from the operational network. That is materially stronger evidence than a firewall-only speed test.

The test does not establish per-hop headroom, firewall CPU consumption, queue depth, application experience at peak times, or failover convergence. Firewall CPU and state-table data were not collected during these runs. Cumulative interface counters showed no reported errors or drops at the later capture, but they do not isolate the benchmark window. Separate routing/failover tests belong to separate verification records.

// the engineering takeaway

For a multigigabit upgrade, keep the chain of evidence intact: confirm the commercial service state, verify physical negotiation, identify every link on the end-to-end path, test from a real workload, use multiple capable external endpoints, retain machine-readable outputs, and interpret loaded latency separately from idle ping. When one test disagrees with the others, document its context before declaring a bottleneck.

The upgrade mattered because the network — not just the WAN socket — could carry the new service rate. The useful result was the evidence trail: a named test path, repeatable measurements, explicit exclusions and honest boundaries around what remained unverified.

// method and evidence note

Measurement source: my validation record dated 24 October 2026, covering tests executed 19 October. Ookla Speedtest CLI 1.2.0, three sequential tests each to Telstra, Superloop and Optus Sydney endpoints; one provider on-net test excluded. Raw JSON and a summary table are retained in internal operations records. Earlier firewall-local iperf3 results are from a separate terminal capture. Internal IP addresses, public WAN addresses, machine identifiers and result URLs have been intentionally omitted.

Six months operating an Akash provider

John Jacob (JJozzieTech) — Thu, 29 Oct 2026 19:28:32 +0000

In May the provider missed a mandatory upgrade deadline, lost its audit attestation, and watched roughly four-fifths of its active leases disappear inside two weeks.

The recovery took twenty-six days. Most of that time was not spent on the upgrade. It was spent on things that had been quietly broken for months — including one bug that had been suppressing all storage reporting for sixty-seven days without producing a single log line.

That turned out to be the pattern. Every expensive incident in the six months since has had the same shape: nothing failed loudly. Something simply stopped happening, and nothing said so.

// what this covers

May to October 2026, on a production Akash provider running in a Sydney enterprise lab. The provider is older than that, but the documented record starts in May, and I would rather write six months I can evidence than twelve I can't.

Specific numbers where I have them. No numbers where I don't. Nothing here is reconstructed from memory — it comes from after-action reviews written at the time, a daily health check that has been running throughout, and the chain.

// the deadline

The provider ran a mandatory upgrade deadline on 1 May and missed it. Audit attestation was removed. Lease count began falling immediately and was still falling two weeks later.

The first recovery attempt lasted about forty minutes. It stopped at the ingress chart install with a version constraint:

chart requires kubeVersion: >= 1.31.0-0
which is incompatible with Kubernetes v1.28.15

The new ingress controller needed Kubernetes 1.31 or later. The cluster was on 1.28.15, which was already end-of-life. There was no flag to pass, no version to pin back to, no way through.

Recovering the audit meant upgrading Kubernetes first. Which meant the cluster had to survive six minor-version jumps. Which meant it needed a control plane that could lose a node, and backups that existed.

None of those were true.

// what the deadline exposed

The honest inventory, severity-ordered, as it stood when the recovery started:

No etcd backups. None. A single disk failure on the control plane's data directory would have meant total cluster state loss. This had been true for sixty-seven days.

One control plane. Any failure of that one node took the whole cluster down. Also true for sixty-seven days and counting.

Kubernetes end-of-life. No security patches, and the hard blocker on the ingress controller that stopped the first attempt.

Storage reporting empty. The provider could not bid on any order requiring persistent storage, because the inventory operator reported no storage capacity at all. Known for sixty-seven days, investigated repeatedly, unresolved.

The deadline did not create any of this. It made it unavoidable. That is roughly the only good thing to say about missing a deadline: it converts deferred work into work.

// foundation

Backups first, because everything after it was going to be destructive.

An etcd snapshot script on a six-hour cron, writing locally and copying off-host, with seven-day retention. Then — and this is the part people skip — a restore tested end to end into a scratch directory, because an untested backup is a belief rather than a backup.

Then two new control-plane VMs, prepped on two separate physical hosts. Not two total: three. A two-member etcd cluster tolerates zero failures and introduces split-brain risk on a network partition, because either member can believe it holds quorum when it can't see the other. Two members is strictly worse than one. The decision that followed from that — join both new control planes in a single session rather than splitting across two mornings — is the one I would most want another operator to take away from this.

The backups paid for themselves within forty-eight hours, not as a recovery but as permission: every subsequent risky step had a fresh snapshot behind it and therefore did not need to be agonised over.

// the hardware interrupt

Three days into the plan, one of the physical hosts bricked. Cause never fully diagnosed — the decision to swap the chassis came before the diagnosis did, because the migration was the priority and the host was replaceable.

Ten days of pause while a replacement was sourced and racked.

One thing came out of it worth keeping: the prepared control-plane VM survived the host swap intact. Its disks moved to the new chassis and it booted with all the preparation work from a week earlier still in place. That pattern — prep the VM, treat the host as fungible — has been reusable every time hardware has moved since.

// measuring the failover

Three control planes joined, etcd at three members, virtual IP with three-way failover. On paper, high availability.

On paper is not evidence. So the next session was spent deliberately breaking it: hard-stopping each control plane in turn at the hypervisor, with no graceful shutdown, and measuring what actually happened.

Test	API endpoint outage	Quorum	Recovery
Primary control plane killed	~0s observed	2 of 3 held	clean, ~90s
Second control plane killed	~8s	2 of 3 held	clean, ~90s
Third control plane killed	~4s	2 of 3 held	clean, ~90s

Under ten seconds in every case. Tenant workloads were unaffected throughout.

Two things worth noting. The first test's apparent zero-second outage was a monitoring artefact — the monitor was running somewhere affected by the test. Where you put the observer matters as much as what you're observing.

And hard-stopping at the hypervisor produces a sharper failure than stopping the service. A graceful shutdown lets everything deregister politely. That is not the failure you are worried about.

I have not seen these numbers published for a Kubernetes cluster this size. They are better than I expected.

// six minor versions in a week

The plan allowed one morning per Kubernetes minor version with soak days in between. Under high availability it was so uneventful that the upgrade ran past its own target and kept going — six minor jumps inside a calendar week, control planes one at a time, workers in batches, cluster never below two functioning control planes.

The lesson is narrower than "Kubernetes upgrades are easy." It is that the upgrade was never the risk. The risk was doing it without a control plane that could lose a member. Once that was true, the thing everyone dreads became routine.

// sixty-seven days

Then the audit submission needed persistent storage, and the storage problem was still there.

The symptom: the inventory endpoint returned an empty storage array. Every other signal said storage was fine. The Ceph cluster was healthy with 19 TiB raw. Persistent volume claims bound normally when tested. The StorageClass carried both documented annotations. The provider simply declined every order that asked for persistent storage, reporting no capacity.

What had been tried across those sixty-seven days: adjusting the annotations, restarting the inventory operator, downgrading it, removing a broken liveness probe, and a complete provider version migration undertaken partly on the assumption that it would resolve. It did not. A post in the community provider channel got no answer.

What broke it was cloning the provider source at the exact version deployed and reading the inventory operator's storage aggregation.

The code discovers every StorageClass, builds a map, and then filters that map before reporting. The filter tests a boolean read from a bare-key label on the StorageClass. Not one of the two documented annotations — those are downstream metadata and gate nothing. A different key, a different metadata type, same namespace.

The StorageClass had both annotations and not the label. So the operator discovered the class, tracked it, and dropped it during aggregation. No error. No warning. Not even a debug line. Just an empty array.

The fix was one command:

kubectl label storageclass <name> akash.network=true

Five seconds later the inventory endpoint reported full pool capacity. Within one bid cycle the provider was bidding on persistent-storage orders again. The audit was submitted the next day and passed all seven benchmarks.

Sixty-seven days. One label. Roughly two hours of actually reading the source.

The lesson I wrote down at the time was: always read the actual source code when a known issue persists across versions. The honest version is the note I added underneath it — should have done this sixty days earlier.

That same bug has since produced two upstream contributions: a documentation issue arguing the fix should be reachable from the symptom rather than only from the install sequence, and a later issue about the silence itself. The operator still drops unlabelled storage classes without saying so.

// hardening

The three months after the audit recovery were spent on things the recovery had exposed as overdue.

Certificates moved to automated renewal — both the provider certificate and the wildcard ingress certificate, with a file-watcher reload path so renewal does not restart the provider. That one has its own write-up and I won't retell it here, but the failure mode behind it is worth stating plainly because it recurs later in this piece: chart defaults silently reassert themselves on every upgrade unless explicitly overridden. That is now a documented pre-flight check rather than something to remember.

Anti-mining enforcement was formalised into three layers and published. The blacklist that drives it is public, with its evidence standards written before the first entry rather than after the first dispute, and with an honest provenance note about which entries carry per-entry evidence and which do not.

And the operational record started being published monthly. The argument for that is simple: a tenant deciding where to deploy has almost nothing to go on beyond on-chain attributes and a price. A provider that publishes its own record — including the parts that went badly — is easier to evaluate than one that doesn't. The secondary benefit turned out to matter more. A monthly report forces a reckoning with the things that are quietly not working, on a schedule, whether or not you feel like having it.

// a silent two weeks

One of the twenty-four nodes held Ready for two weeks with no working network path for its pods.

Kubernetes said the node was fine. The kubelet was reporting heartbeats on schedule, the pods assigned to it showed as scheduled and healthy, nothing was being restarted. The signal Kubernetes watches is its own control-plane connectivity to the kubelet, which was intact throughout.

What was broken was the data-plane. Pods on that node couldn't reach storage, couldn't reach each other, couldn't reach anything outside — and the signal that would have shown it never travelled through the kubelet, because the kubelet wasn't the thing broken.

It was caught by cross-referencing two independent sources of truth: the Kubernetes node list against the storage cluster's own view of which hosts its daemons were reachable from. A node that holds Ready on one side but is unreachable from the other is in a split state, by definition, and that cross-reference is now one of the handful of checks that run on a schedule.

Nothing had alerted during the two weeks. The failure mode was the shape every expensive incident in these six months carried — a thing that stopped being true, with nothing in the stack built to notice.

// the rebuild

August and September were physical.

The provider estate had been running on a single-core legacy switching design. One switch failure took the entire provider. Any change to that core meant a full outage, which is why several improvements had stayed deferred for months — the cost of touching it was always higher than the cost of not touching it, right up until it wasn't.

The programme ran in three phases over a month: relocating the small-form-factor node estate to new rack space, then the hypervisors one at a time, then the network cutover itself — twenty-four hosts moved from the legacy gateway to a redundant gateway pair on a dual-TOR fabric, workers first, control planes second, the primary control plane last. One near-miss from the broader rack programme — on the validator side, not the provider side — has its own write-up.

Zero leases lost across the entire programme. Twenty-four nodes, twenty storage daemons, three etcd members, three storage monitors, all relocated.

Three things from that month I would not want to rediscover.

noout is the wrong tool for a rack. It is the obvious-looking shortcut for a maintenance window — tell the storage cluster not to rebalance while hosts are down. For a single host, briefly, it is correct. For multiple hosts under a three-replica policy with host-level failure domain, it would have left roughly forty percent of placement groups inactive. The obvious shortcut and the correct action diverge sharply at scale.

A running component can be in an unrecoverable state without being unhealthy. A GPU operator had applied a scheduling taint to one workstation. A storage monitor and a storage daemon were running there perfectly — and could never have been rescheduled if they stopped. Nothing alerted. It surfaced only when I tried to move them deliberately. Healthy and recoverable are different properties and only one of them is monitored.

Diagnose from state, not from logs. Twice in the programme a storage daemon was diagnosed from heartbeat log text, and twice it was the wrong daemon. The actual fault both times was a node losing power to a failing cable. One command listing daemon state gives the answer directly. That is now the first command rather than the third.

There was also a switch whose running configuration had never been written to startup — a reload would have taken the configuration with it — and two marginal ports with around twenty thousand accumulated receive errors, which on recabling retrospectively explained three unattributed incidents from earlier in the year. Intermittent physical faults present as software mysteries for as long as you let them.

// four days of nothing

During the network cutover the provider created no bids for four days.

It recovered without intervention. No leases were lost — existing workloads ran throughout; only new bidding stopped. Nothing alerted, because nothing had failed. There was no error to catch.

It was found afterwards, by counting transactions.

This is the part worth sitting with. A provider that has stopped bidding looks identical, on every dashboard available to its operator, to a provider that is bidding and simply not winning. Both show healthy pods, healthy storage, healthy nodes, active leases ticking over. The difference between them is a thing that is not happening, and monitoring is built to catch things that happen.

The same month's provider upgrade revealed a longer-running version of the same problem. Three previous upgrades had failed outright — and the provider had kept running on its last good configuration the entire time, looking healthy by every external check. Worse, each subsequent upgrade correctly built on the last successfully deployed configuration, silently discarding everything in the failed ones.

Six weeks of changes that appeared to have been applied and had not. A certificate setting that never actually changed. An on-chain attribute update that never deployed. Two values that stayed inverted on-chain until they were corrected directly.

Both root causes turned out to be upstream bugs in the deployment chart — one where disabling a feature drops the container's entire environment block and the resulting object is rejected, another where an attribute comparison sorts keys and values independently and therefore cannot see two values that have been swapped. Both are now filed.

// unaudited

In August the attestation was removed again — this time not through any failure on the provider's side. An auditor adjusted the scope of its portfolio and this provider fell outside it. No compliance finding. Nothing about the hardware, the uptime or the pricing changed.

The practical consequence was larger than expected, and took an outside dataset to make visible.

The deployment console's default flow filters to audited providers. So does every trial account. The effect was immediate invisibility to the main discovery path, with no signal on the provider side at all — because the way that invisibility manifests is a bid that never happens.

An independent benchmarking project noticed first. It had been running automated deployments across the network for a month and observed that this provider had stopped bidding on its orders entirely: hundreds of orders broadcast, hundreds of bids drawn from other providers, none from here — while its own market snapshots showed this provider advertising available capacity in one hundred percent of over a thousand samples across the same window.

From inside, there was nothing to see. The bid engine was healthy and transacting normally. Thousands of bids over the period, a failure rate well under one percent, nearly all of those losing a race to the chain's bid cap.

The answer was on-chain. Pulling a live order from that tenant showed a signed_by requirement naming a specific auditor. Every open order from them carried it. Without the attestation, those orders are unmatchable at protocol level — and the bid engine declines, correctly, every time, in silence, because a declined bid writes nothing anywhere.

There is a reciprocal finding in that, which the other party has since accepted and is fixing: their own tooling had stated publicly, more than once, that it did not filter on audit status. On-chain, it did. The requirement had been inherited from a template rather than chosen, and was invisible from inside their own tooling for exactly the same reason the missing bids were invisible from inside mine — both of us were reading our own output rather than what the chain actually said.

At the time of writing the provider is still unaudited. A re-audit request was submitted in October and is outstanding. The connectivity-upgrade dependency that had previously held the request — a WAN capacity upgrade from Optus 800/80 to Aussie Broadband 2000/500 Mbps — has since completed. What remains is the audit response itself.

// the pattern

Four incidents cost real time or real revenue across these six months.

A storage class silently filtered out of inventory for sixty-seven days. Deployment upgrades that failed while reporting success for six weeks. A node that held Ready for two weeks with no working network path. Bids that simply never happened.

Not one of them produced an error. Not one would have been caught by alerting on failures, because none of them was a failure in any form a monitoring system recognises. They were absences — and an absence does not write a log line, does not increment a counter, and does not fire an alert.

What has actually worked against this is narrow and unglamorous. Count things that should be happening and alert when the count is zero, rather than alerting when something breaks. Cross-reference two independent signals and treat divergence as a fault, which is how a silently partitioned node is now detected — with the honest caveat that the check currently covers nineteen of twenty-four nodes, because five carry no storage daemon to cross-reference against, and that gap is still open.

And publish the operational record. Two of the four were found because an outside party was measuring, and one of those was found because they published their measurements and I could check them against my own. That cuts both ways — the monthly reports exist so somebody can do the same to me.

That is the uncomfortable conclusion of six months. The failures that cost the most were not the ones that broke. They were the ones that stopped — and nothing, anywhere, said so.

// the receipts

Everything above is checkable. The repositories are sanitized but not curated — what's in them is what runs.

akash-provider-ops-public — operational scripts, monitoring and anti-mining tooling, and the monthly operator reports under reports/
akash-miner-blacklist — the community blacklist, its evidence standards, and its provenance disclosure
Upstream — the storage-label issues (#645, #663) and the two deployment-chart bugs (#595, #596)

The operator reports are published monthly and carry the same structure: what ran, what shipped, what broke. They include the months where the answer to the third one was long.

Operated from Sydney, au-syd region.

iSCSI gives you two minutes

John Jacob (JJozzieTech) — Thu, 22 Oct 2026 19:42:58 +0000

You can have enterprise storage, a healthy host and a filesystem doing exactly what it was designed to do, and still lose a node to one network interruption. This is what a firewall racking job did to my Sia hostd node, and what it taught me about how much interruption iSCSI will actually absorb before the filesystem underneath runs out of options.

It started with something that looked like a failure and turned out to be me misreading my own setup. Twenty-four hours later it became the real thing.

// the setup

The hostd node runs as a VM on pro4, one of the Proxmox hosts, on the workload VLAN. Storage comes from TrueNAS over iSCSI — a zvol presented as /dev/sdb, formatted XFS, mounted at /mnt/sia-data, with /var/lib/hostd symlinked into it. hostd itself is a single binary handling contracts, storage proofs and peer connections.

iSCSI over a routed network is convenient. It is also unforgiving. Convenient because the disks live where the disks live, the compute lives where the compute lives, and neither has to know. Unforgiving because the amount of interruption it will tolerate is a fixed number in a config file, and most of us have never looked it up.

I hadn't.

// act 1 — the false alarm

The morning after the migration, the hostd GUI wouldn't load. My first assumption was a crash, and the logs backed it up — a startup attempt with failed to create data directory, which reads alarming at seven in the morning.

It took about thirty seconds to establish I was wrong. hostd was already running. Process up, port 9980 bound and listening. My systemctl start had found a live process and exited without doing anything. The GUI had been reachable the whole time; I had the browser pointed at the management VLAN address rather than the workload one.

So: operator error, not a node failure. Lesson one, filed and then almost immediately forgotten — verify before you assume.

// act 2 — the real one

Twenty-four hours later I was racking the new border firewall. During the physical install it was down for roughly twenty minutes. That was planned and I was expecting it. What I had not fully accounted for was that the iSCSI path to TrueNAS ran through it.

The sequence:

The network path to TrueNAS dropped when the firewall went dark.
Five seconds later, with no reply to its NOP-out ping, the initiator declared the connection failed.
The session went into recovery and I/O queued, waiting for the session to come back.
Two minutes in, it hadn't. SCSI commands were failed back to the application and /dev/sdb went away.
XFS saw write errors on a device that no longer existed and shut down the journal to protect the filesystem.
hostd was mid-write. database disk image is malformed on hostd.sqlite3.

journalctl told the whole story in two lines — write I/O errors, followed immediately by XFS shutting itself down. hostd had crashed cleanly enough, but the SQLite write-ahead log was in an uncertain state and the database was 41 GB.

// the two numbers I should have known

Here is the part I had wrong, and had been wrong about for as long as I'd been running this.

I assumed iSCSI had no tolerance at all — that a dropped path meant an immediate dead device. It doesn't. There are two timers, they do different jobs, and the gap between them is where the fault tolerance lives:

node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout  = 5
node.session.timeo.replacement_timeout = 120

Those are the stock defaults, unchanged on my host. The initiator pings the target every five seconds. If a ping goes five seconds without a reply, the connection is declared failed — that's the first number, and it's the one I'd half-remembered as "the timeout".

It isn't the one that matters. replacement_timeout is how long the session is given to re-establish itself before SCSI commands get failed back to the application. Default 120 seconds. For two full minutes, I/O queues and waits. If the path returns inside that window, the writes land and nothing upstream ever knows there was a problem.

So iSCSI is not fragile. It ships with two minutes of patience.

I was gone for twenty.

That reframes the incident entirely. Nothing failed — not the initiator, not XFS, not SQLite. Every layer did exactly what it was built to do, in order, on a timescale that was documented and that I could have read in thirty seconds any time in the previous year. The outage was ten times longer than the tolerance, and after that there is nothing left for a filesystem to do except stop.

// the recovery

Methodical, and mostly waiting. Once the firewall was back and the path restored:

umount -l /mnt/sia-data

Lazy unmount, then remount — XFS replayed the journal automatically on the way back up.

sqlite3 /var/lib/hostd/hostd.sqlite3 "PRAGMA integrity_check;"

Then wait. On a 41 GB database the integrity check ran for about three hours. The result, when it finally came back, was one word: ok.

systemctl start hostd. The GUI came up, contracts resumed, and RHP4 RPC successes started streaming in the logs within minutes. Nothing was lost.

// what I'd make instinctive

Three things from this.

Stop the service and unmount before touching anything on the storage path. Routing changes, firewall reboots, link reseats, a switch upgrade — if the path between the VM and the iSCSI target is in scope, hostd stops and /mnt/sia-data unmounts first. It costs a minute. The alternative cost me three hours of integrity check and a morning of not knowing whether the database had survived.

Know your replacement_timeout before you need it, not after. Two minutes is the default, not a law — it's one line in iscsid.conf and some distributions and appliances ship it differently. Run iscsiadm -m node -o show | grep -E 'replacement_timeout|noop_out' on anything you care about and write the answer in the runbook next to the maintenance window. A twenty-minute job against a two-minute tolerance is a decision, as long as you know you're making it.

SQLite's WAL is resilient, but only because XFS was. The write-ahead log protected the database and the XFS journal protected the filesystem. Both did their jobs. The three-hour integrity check wasn't a symptom of damage — it was the cost of confirming the safety nets had held, on a database large enough that confirming takes three hours.

The runbook now has a mandatory pre-change step: stop hostd, unmount the iSCSI volume, then proceed. Every time.

// closing

The ARP race piece from last week was the same shape at a different layer — a change that verified fine at one point and broke at a boundary nobody had checked. This one is the same lesson with a number attached. The maintenance window was planned, the outage length was known in advance, and the piece of information that would have made it a non-event was sitting in a config file I'd never opened.

The tolerance exists. It's just shorter than most maintenance windows, and nothing warns you when you're about to spend it.

The storage failure that was actually an ARP race

John Jacob (JJozzieTech) — Thu, 15 Oct 2026 19:42:11 +0000

The Windows workload client lost its TrueNAS shares one Thursday morning. Red-X on every drive mapped to that specific server. Mappings to other storage servers on the same VLAN stayed online. Test-NetConnection on port 445 returned TcpTestSucceeded : False. Ping timed out.

I cleared the client's ARP cache. The shares came back. Then Windows Update rebooted the client, and the shares went down again.

That second failure is what made me look properly, because the first failure had a satisfying workaround and the second one broke that workaround in a way that was going to keep breaking it. If I'd stopped at arp -d, I'd have carried a time bomb forward — Windows Update reboots are routine, and every reboot would have been a 50/50 bet on whether the shares came back.

This piece is the trace that turned a storage symptom into a network root cause: client to Nexus to endpoint, with each layer eliminating a correct-looking suspect, ending in a configuration error that had been in place since the fabric migration. The server I thought was failing wasn't failing. Nothing was failing. Two devices were simply configured with the same IP address, and ARP — behaving exactly as designed — had been quietly handing my client the wrong one.

// why storage was the first suspect

The affected TrueNAS server had been through genuine storage trouble recently. The MMX/TrueNAS migration piece documented five layers of failure during that same ReRACK window, including a hung SAS/CAM path on this same server that required a full JBOD power cycle to clear. When storage had just bitten me, storage was the natural place to look.

The symptoms were plausibly storage: SMB mappings timing out, Red-X on the shares, retries not helping. If this had been a repeat of the earlier SAS failure, I'd have expected pool-level trouble visible in the TrueNAS GUI, SMB service errors in the system logs, or at minimum a kernel message about a hung path. None of that was there. Pools online. SMB service healthy. Other storage servers on the same VLAN reachable from the same client, serving the same workload, with no complaints.

That mismatch — storage-looking symptom with no storage-layer evidence — was the first thing that didn't fit. Storage went to the back of the suspect list within about ten minutes. The symptom was Layer 7, but the fault was somewhere lower.

// the Windows ARP red herring

The next suspect was the client itself. SMB has a long history of edge cases on Windows that can be fixed by clearing some piece of cached state and letting the stack rebuild. The ARP cache is one of them. If the client has learned a stale MAC address for the server, nothing above Layer 2 is going to work until that mapping refreshes.

I ran:

arp -d 172.16.x.22
ping 172.16.x.22
arp -a | findstr 172.16.x.22

(Addresses are illustrative throughout this piece — I'll use the 172.16.x.NN pattern to abstract real addressing.)

The ping succeeded. The ARP cache populated with a MAC. The SMB mappings came back online. Everything felt resolved.

For a reader who's spent any time troubleshooting SMB on Windows, this is the moment that feels like victory and is often a trap. arp -d restoring service is a specific signal — it says the client was holding the wrong MAC for the IP it was trying to reach. It does not say why the client was holding the wrong MAC. There are at least three plausible reasons, and only one of them is a Windows bug.

I didn't think about which reason it was. I called the shares back up, verified the mappings, and moved on. For about two hours it felt like I'd fixed something.

Then the client took a Windows Update reboot. The ARP cache cleared as a side effect of the restart. Windows did a fresh ARP lookup on the TrueNAS address. The SMB mappings came up Red-X again.

That recurrence is what forced the real investigation. A workaround that works once can be a workaround. A workaround that works and then breaks on the next reboot is pointing at something underneath that nobody's looked at yet.

// tracing from client to the fabric

The question I needed to answer first was: when the client sends traffic to the TrueNAS address, where does the fabric think that traffic should go?

The TrueNAS server runs as a Proxmox VM, which attaches to the Cisco Nexus 3548 top-of-rack pair via a vPC-bonded port-channel on the hypervisor. The TrueNAS MAC should appear in both TORs' MAC address tables, learned on that port-channel.

I checked:

TOR-A# show mac address-table address MAC-A
+  10  MAC-A    dynamic   0  ...   Po200

TOR-B# show mac address-table address MAC-A
+  10  MAC-A    dynamic   0  ...   Po200

(MAC addresses abstracted as MAC-A, MAC-B, MAC-C throughout.)

Both TORs had the TrueNAS MAC on Po200, which is the LACP bundle to the correct Proxmox host. The vPC was consistent, the trunk allowed the workload VLAN, and port type was edge trunk as it should be. From the switch's perspective the TrueNAS server was exactly where it was supposed to be.

The fabric wasn't the problem. The fabric was doing the thing fabrics do: forwarding traffic to the MAC that owns the destination IP, as that MAC resolves through ARP, as the ARP table currently believes.

Which meant the real question was who owns the destination IP in the ARP table, and the answer to that was about to be the whole story.

// the ARP table discrepancy that unlocked everything

I checked the gateway ARP table on the Nexus for the TrueNAS address:

TOR-A# show ip arp vlan 10 | include 172.16.x.22
172.16.x.22    00:00:44    MAC-B    Vlan10

The MAC in that line was not MAC-A. It was a MAC I didn't recognise — MAC-B. For an address I thought belonged to TrueNAS, the gateway's ARP table was returning the MAC of some other device.

This is the moment the investigation shifts. Up until this point I'd been looking for things that were broken — a storage pool, an SMB service, a Windows cache. Nothing was broken. Everything was working. What was happening was that the fabric was routing .22 traffic to a MAC that was not the server I thought owned .22, and it was doing that because that MAC had answered the gateway's most recent ARP request for the address.

I traced the unfamiliar MAC through the CAM table:

TOR-A# show mac address-table address MAC-B
*  10  MAC-B    dynamic   0  ...   Eth1/15

Eth1/15 on TOR-A. The port profile named it — a different workload node on the same VLAN. Not Proxmox, not TrueNAS. A separate storage/compute node attached directly to one of the TORs.

For completeness I checked the client's path too — its MAC resolved on TOR-B Eth1/21, the port profile matching the expected workload-client access port. Clean. The client was correctly learned, in the correct port, in the expected VLAN. No fabric-side problem on the client's side either.

The whole fabric was behaving as designed. What wasn't right was higher up: two different MAC addresses were able to answer ARP for the same IP, and the fabric had no way to know which one was supposed to.

// the endpoint reveal

The authoritative check for IP ownership is the host itself. The switch can tell you where a MAC lives. Only the endpoint can tell you what address it thinks it has.

I logged into TrueNAS directly and ran ip addr show:

3: enp6s19: <BROADCAST,MULTICAST,UP,LOWER_UP> ...
   link/ether  MAC-A
   inet  172.16.x.22/24  brd 172.16.x.255  scope global enp6s19

TrueNAS was configured as .22. The other workload node was configured as .22. Both devices had legitimately claimed the same address. Both were valid Layer-2 participants on the workload VLAN. Both would answer an ARP request for .22 with their own MAC.

The intended address for TrueNAS during the ReRACK was .40. I had the record of the planned allocation; .22 belonged to the other node and had belonged to it for months before the fabric migration. What had happened was that during the TrueNAS reconfiguration for the new fabric, the address .22 had gone onto the interface by mistake — one digit off, or a copy from a different line of a planning spreadsheet, or anything in that family of five-second configuration slips. Nobody caught it at the time, because .22 on the TrueNAS interface produced a device that answered ARP and accepted TCP connections. From any single-request check, it looked fine.

It only became a problem when both devices were on the VLAN at the same time and the client needed to decide which one owned .22. The client couldn't decide. The client just believed whatever ARP reply arrived first.

// what the duplicate IP actually did

ARP is one of those protocols where violating the design assumptions can stay quiet for a long time. Nothing I was watching logged a complaint — no pool errors, no SMB service faults, nothing in the fabric. NX-OS does have duplicate-address detection, and it never surfaced this one; with the second node mostly idle on that address, the two never ARPed close enough together to trip it. The vPC pair stayed consistent, HSRP didn't flap, MAC flapping didn't trigger because the two MACs were on different ports and neither of them was moving.

What the duplicate IP did was turn ARP resolution into a race. When any device on the VLAN sent an ARP request for .22, both TrueNAS and the other workload node answered. Whichever answer arrived first became the entry in the requester's ARP cache. The entry held until it timed out or the cache got cleared, at which point another race ran.

For the client, this looked like intermittent connectivity. Half the time the cache held TrueNAS's MAC and SMB worked. Half the time it held the other node's MAC, traffic went to a host that had no SMB listener on that address, and the shares came up Red-X. The 50/50 wasn't literal — timing depends on where the devices sit relative to the client, which replies win in which order — but the practical effect was that the client sometimes worked and sometimes didn't, with no deterministic rule for which.

arp -d restored service because it forced a new race. If the race landed on TrueNAS, the shares came back. If it landed on the other node, nothing changed. For my situation, the first few arp -d calls happened to land on TrueNAS, which is why the workaround felt like a real fix. It wasn't fixing anything. It was rerolling the dice, and I happened to roll well for a few hours.

The Windows Update reboot was what exposed this. A reboot clears the entire neighbour cache, which forces a fresh resolution. The post-reboot resolution landed on the other node this time, and the shares were Red-X again. That recurrence was the lucky break — if the reboot had landed on TrueNAS, I'd have gone another day or two before the next failure, and the pattern would have been even harder to see.

Clearing ARP wasn't a workaround for a Windows bug. Clearing ARP was the trigger I was using to reroll an unstable system state, without understanding that the state was unstable.

// the fix

With the duplicate IP identified, the fix was configuration-level and non-destructive. Both devices were legitimately part of the VLAN. The other workload node had owned .22 for months and was the historical owner. TrueNAS was the one with the mis-assigned address.

I verified that .40 was unused on the VLAN, by checking the gateway's ARP table for the address and getting no entry. Then changed the TrueNAS interface configuration in the GUI from .22/24 to .40/24, leaving everything else — the MAC, the subnet mask, the gateway, the DNS settings — unchanged.

On the client side, the persistent drive mappings had to point at the new address. The net use entries for the two affected shares came off and went back on with the new target:

net use G: /delete
net use H: /delete
net use G: \\172.16.x.40\JBOD1 /persistent:yes
net use H: \\172.16.x.40\JBOD2 /persistent:yes

Both drives came up on first attempt. SMB access was clean, reproducible, and stable across the next series of planned client reboots. The duplicate-IP condition was gone because only one device now claimed .22, and that device was the one that had always been the legitimate owner.

The whole corrective action took about ten minutes once the root cause was identified. The investigation that identified it took the better part of a morning.

// what I'd make instinctive

Four things from this.

When clearing ARP "fixes" a problem, test for duplicate IP before treating the client as buggy. arp -d is a legitimate diagnostic tool, but a successful ARP clear doesn't prove the client's ARP implementation is broken. It proves the client was holding one of several possible mappings, and the one it was holding wasn't the right one. There are three plausible reasons for that: the client had learned a stale MAC that no longer owned the IP (an actual client-side cache problem), the switch had learned stale CAM entries for the MAC (a fabric-side cache problem), or two devices were legitimately answering for the same IP (a configuration problem). The first two are rare on modern hardware with healthy aging timers. The third is more common than operators like to admit, and it's the one that arp -d can't fix because the next ARP exchange has the same race condition.

ARP tables answer IP → MAC. CAM tables answer MAC → port. Use both directions. The decisive moment in this investigation was comparing what the gateway ARP table thought owned the IP against what the CAM table said about where that MAC lived. The ARP table alone would have told me the resolution was wrong but not where the wrong device was. The CAM table alone would have told me a MAC was on a port but not that it was answering for an IP it shouldn't own. Both tables, read in the same session, pointed at the specific switch port on the specific TOR that was serving the wrong endpoint. That's not something you can see from either table in isolation.

The switch tells you where a MAC lives. Only the endpoint tells you what IP it thinks it owns. This is the kind of lesson that sounds obvious when written down and gets forgotten under time pressure. During the investigation I was tempted to stop at "wrong MAC answering for the IP, that's the problem" and go hunt down the owner of that MAC in the inventory. The step that actually resolved the question was logging into TrueNAS and running ip addr show — that's what proved the mis-assignment at the authoritative source. Any investigation that ends at the switch is incomplete; the switch is reporting observations, not declarations of ownership.

Static ARP is the wrong permanent answer to duplicate IP. It's tempting, once a duplicate IP is identified, to pin the correct mapping on the affected client with a static ARP entry. That hides the symptom for one client. It leaves the duplicate IP in place for every other device on the VLAN — the gateway, the DHCP server, the monitoring stack, any other client that happens to need the address. The correct action is to fix the IP ownership at the source, which is a configuration change on the mis-assigned device. Static ARP as a durable fix means you've accepted that your VLAN has a duplicate IP and chosen to paper over it on the one host that noticed.

None of these are theoretical. All four came out of this specific morning's investigation, and all four are things I'd apply next time I see a symptom that arp -d temporarily resolves.

// closing

The Article 10 rack migration piece landed on the observation that a plan can be right and still be incomplete — correctness and completeness are different axes. This piece is a smaller version of the same shape at the IP layer: a reconfiguration had been done, had been verified to the extent that single-point checks could verify it, and had been complete enough that TrueNAS came up on the VLAN and started answering queries. What the single-point check couldn't see was whether anyone else on the VLAN was also answering for the same address. That verification needs the full ARP table, not just a ping to confirm the host is reachable.

If you have anything on your network that "sometimes needs an ARP clear to come back," you probably don't have a Windows ARP bug. You have a duplicate IP condition that hasn't been investigated yet, and the ARP clear is masking it by rerolling the race. Investigate the next time the symptom appears, not the next time it's convenient — the next client reboot is going to reroll the dice regardless, and sooner or later it's going to land in a way that costs more than one morning.

Four lines of patched JavaScript, and the router VM that deleted them

John Jacob (JJozzieTech) — Thu, 01 Oct 2026 22:45:24 +0000

Two small patches to a dependency file called ssdp.js. Four lines of changed JavaScript, total. They kept the fleet's nodes reachable from the internet, and they were the reason every incident on the fleet, for months, started with the same sentence: "the patches are gone again."

The patches lived inside a Node.js dependency called @runonflux/nat-upnp. Every time the application updated itself — and it updated itself often — the file was rewritten, and the four lines vanished. A cron job would notice within a minute and put them back. A @reboot timer paired with a sleep would apply them again during startup. A watchdog on the edge firewall restarted miniupnpd every two minutes because the UPnP daemon had a habit of losing state without any log line indicating it had. And a verification pass every thirty minutes checked that the patches were still on disk and that the UPnP mappings the daemon had negotiated were still in place.

That's four distinct pieces of automation running continuously to keep four lines of JavaScript alive on seven nodes.

The migration's purpose was not to improve any of that. It was to delete it.

// why the patches existed

The edge firewall running miniupnpd was configured correctly. The UPnP daemon was up, listening on the LAN interface, advertising the Internet Gateway Device profile the nodes expected. And it refused to talk to any of them.

The problem was M-SEARCH. UPnP clients discover their gateway by sending an M-SEARCH message to a multicast group on 239.255.255.250. That works when both endpoints are on the same L2 segment and the multicast group is joined on both sides. On this network, the firewall's LAN interface never joined the group. From the daemon's side everything looked fine; from the nodes' side the M-SEARCH went out and nothing came back.

The first patch made the nodes send M-SEARCH as unicast to the firewall's LAN address instead of to the multicast group. Two lines of change to the SSDP client code inside nat-upnp. Once applied, discovery worked immediately. The daemon answered the unicast M-SEARCH, the node completed its discovery step, and the port mapping request went through.

The second patch was for a different problem. nat-upnp binds to every network interface on the host during socket creation. On a FluxOS node running Docker, that includes Docker's 172.x bridges — interfaces that have their own UPnP behaviour (specifically none) and cannot support the port mapping the client tries to negotiate. The socket creation attempt against those interfaces returned UPnP error 718 — "conflict in mapping entry" — and the client crashed before it could try the interface that would actually have worked.

The second patch filtered the socket creation loop by interface, letting nat-upnp skip anything that didn't look like a routable interface pointing at the gateway. Two more lines. Once applied, socket creation succeeded on the first attempt.

Two patches, two specific bugs, four lines of code between them. Both trivial. Both load-bearing.

// the maintenance tax

The problem with load-bearing patches to a third-party dependency is that the dependency doesn't know about them. nat-upnp shipped through the application's normal update cycle. Every application update pulled a fresh copy of the dependency, which overwrote ssdp.js with the upstream version. The patches were gone in the moment of the update. Discovery would fail on the next M-SEARCH, port mappings would drop shortly after, and the node would fall out of the fleet's reachable set.

The maintenance response was a stack of automation that grew a piece at a time as I learned what the failure modes were:

A reapplication cron on a fixed timer. Every minute, check the current ssdp.js against the patched version. If they differ, apply the patches, restart the client. This caught the majority of updates within sixty seconds of the file being rewritten.

A @reboot timer paired with a startup sleep. The one-minute cron didn't fire until the node had been up for a minute, and the application started faster than that on some hosts. Without the reboot-time apply, the client could initialise against the upstream ssdp.js, cache its (broken) module state at load time, and continue running with the wrong code for the first minute after every boot. The @reboot sleep 30 && apply-patches entry closed that gap.

A watchdog on the firewall itself. miniupnpd had a habit of drifting into an unresponsive state — not crashed, not exited, just no longer answering M-SEARCH. A systemd unit on the firewall restarted the daemon every two minutes, unconditionally, so that any drift was bounded to the same two-minute ceiling. This was crude and honest — there was no signal I trusted to detect the drift, so the watchdog restarted the daemon on a schedule instead.

A re-verification pass every thirty minutes. Both cron and the boot timer could fire successfully and still leave the fleet with broken port mappings, if the daemon had lost state during a window when the patches happened to be in place. The verification pass ran the equivalent of upnpc -l from each node against the firewall, checked the returned mapping list against the expected mappings, and re-negotiated any that were missing. This was the safety net for the cases the other three mechanisms didn't catch.

Four distinct pieces of automation. Two of them local to the nodes (cron and @reboot), two of them running on the firewall (miniupnpd watchdog and the verification pass). All four load-bearing. If any one of them stopped running, the fleet would degrade within an hour.

I don't want to make this sound heroic. It wasn't. It was a workaround for a bug that a proper fix would have resolved once and stayed fixed. The workaround happened to work — for months — and every month I told myself I'd do the proper fix during the next quiet week. There was never a quiet week.

// why it kept breaking anyway

Even with all four pieces of automation in place, incidents still landed on the fleet at a rate of about one per week. The pattern was always the same: application auto-updates in a way that races the one-minute cron, the client instantiates against the upstream ssdp.js, the module cache locks in the broken code path, and the cron's later patch-apply doesn't help because the running process has already loaded the module.

That last part is the point. Node.js caches modules at first require(). Patching ssdp.js on disk after the client has already loaded it doesn't affect the running process — the module remains resident in memory in its original form until the process restarts. So the reapplication cron only worked if it ran before the client instantiated the module. On a boot, that meant winning a race against the application's own startup. On a mid-life update, that meant restarting the client immediately after re-applying the patches. Both were possible; neither was reliable enough to be trusted at fleet scale.

There was also a doctrine issue that made the problem harder than it looked. Standing rule five on this fleet is: never patch files under ZelBack/src/. Those files carry an integrity hash the application checks at startup. Patching them causes the node to refuse to start at all — a much worse failure than a broken UPnP client. nat-upnp sits outside ZelBack/src/, which is why the patches were possible in the first place. But it meant the workaround had to live in a boundary layer between the application and its dependencies, patching upstream code that could be replaced at any moment by the application itself.

The whole arrangement was a stack of workarounds on top of workarounds. Nothing about it was going to become stable by adding more mechanisms.

// the alternative

The migration's answer was to move the IGD off the nodes entirely.

A router VM sits L2-adjacent to the fleet on the same segment. It holds the public IP address for the fleet through 1:1 NAT — the router owns the public address, the fleet's nodes hold private addresses, and traffic bound for the public address is translated to the router's LAN address and forwarded onto the segment. The router runs its own miniupnpd, correctly configured, joining the multicast group on the LAN side, answering M-SEARCH from the nodes without patches or workarounds.

From the nodes' side, this looks like a normal gateway. nat-upnp unpatched, upstream, running against the vendor's own release. Discovery works because the router VM's miniupnpd correctly joins the multicast group. Socket creation works because the router VM is the only interface pointing at the internet from the nodes' perspective — Docker's 172.x interfaces are still there, but the client's discovery finds the router's LAN address as the gateway of record and doesn't try the Docker bridges.

The router VM approach dissolved every piece of the workaround in one move:

Both patches to ssdp.js — deleted. The nodes run stock @runonflux/nat-upnp and have since the migration.
Three cron entries — the reapplication timer, the @reboot sleep pair, and the verification pass every thirty minutes — all deleted. Nothing to reapply, nothing to re-verify.
The firewall watchdog restarting miniupnpd every two minutes — deleted. The router VM's miniupnpd runs stable because the underlying configuration is correct.
The old DMZ 1:1 NAT for Node 1 — deleted. Node 1 had a special-case NAT rule for historical reasons; the router VM's uniform 1:1 NAT covered it too.

Seven nodes on stock dependency code. No cron entries related to the fleet's UPnP layer. No firewall watchdog. No re-verification pass. And no more one-per-week incidents traceable to the patch treadmill, because there was no treadmill to trace them to.

The lesson isn't that the patches were wrong. The patches worked, for what they were, for as long as they had to. The lesson is that patches designed to survive on top of a moving target aren't going to, and any workaround that has a maintenance tax attached to it is going to fail eventually — not because the workaround is bad, but because the tax compounds.

The right answer wasn't a better patch. It wasn't a better cron. It was an architecture that didn't need either.

// the coda that closed everything else

I want to be honest about how this piece was supposed to end, and how it actually ends.

The piece I wrote in my head, during the router VM build, ended at the previous section. Seven nodes on stock code, workaround deleted, treadmill ended by operator design. A clean win — a workaround built through operator effort, retired through operator effort, on operator terms.

That's not how the story closed.

Since the migration, the protocol has released ArcaneOS. Every node in the fleet is expected to be running it. ArcaneOS is a self-updating, encrypted-disk, no-root, restricted-shell operating system image. It is not the sort of thing you patch files on. The router VM approach continues to work because it lives outside the nodes — the router isn't a Flux node, so ArcaneOS doesn't touch it. But every piece of node-side infrastructure I built or configured over the last year, including the deep operational knowledge of how to keep FluxOS behaving, is no longer mine to work on. I don't have root. I can't inspect the filesystem beyond what the restricted shell exposes. The OS updates itself on its own schedule and I find out about the update after it's done.

That's the coda. The patch treadmill stopped when I moved the IGD off the nodes. It also stopped, more completely, when I stopped owning the OS.

I want to be careful how I write that. It reads two ways. The relieved reading — no more maintenance tax, no more four-in-the-morning patch races, no more workarounds on workarounds — is real. There is genuine relief in never having to reapply a patch to ssdp.js again, because it can't happen. The restricted shell prevents it.

The other reading is also real. Operating a fleet of nodes involves a hundred small decisions about how the OS runs — cron cadence, log rotation, disk pressure alerting, monitoring hooks, package versions, kernel parameters. Every one of those decisions is now made by the protocol, not by me. I have opinions about some of them. Those opinions no longer matter.

The two readings are not opposed. Both are honest. But if I had to pick which one the closing sentence should carry, it's the second one, because the first one is already implicit in "the treadmill stopped." Saying only that would suggest that the ending was clean. The ending wasn't clean. Something ended that I had built, and something ended that I wanted to keep building.

The treadmill stopped when I stopped owning the OS. Both things are true at once.

A coming piece covers the mechanism that made that happen — how the protocol enforcement rolled out, what "every node in the fleet went dark" actually looked like from an operator's seat, and what the operator response was. This piece is the workaround story. That piece is the governance story.

// what I'd make instinctive

Four lessons from the treadmill months.

Patch treadmills mask their true cost until you stop. While the four mechanisms were running, I could tell you what each one did but not what the total cost was. Only after deleting all four could I see how much operator attention had been going into keeping them working. If you have a workaround with a maintenance tax attached, the cost you can measure — the CPU time, the disk writes, the log volume — is not the real cost. The real cost is the operator attention that isn't going elsewhere.

Auto-update-then-patch is a race the auto-update wins eventually. Any workaround that depends on being reapplied faster than upstream updates is going to lose that race, at some point, at the worst possible moment. The correct fix is architectural — remove the need for the patch — not a faster cron.

When you build a workaround, ask what would let you delete it. The router VM approach was possible from day one. I didn't do it because the workaround was working and building the router VM was more work than reapplying patches. Every month that passed, the calculation stayed the same and the workaround stayed. The trigger for the architectural fix should have been the workaround's existence, not the workaround's eventual failure at scale. The /etc/environment piece landed on the same pattern from a different angle on this fleet — a mechanism that had been running for months, wasn't doing the job I thought it was, and needed replacing at the architectural layer rather than fixing at its own. Different failure mode, same operator lesson.

Operator ownership and operator maintenance are the same thing. The ability to run a workaround like the patch treadmill is the same ability the protocol has now taken away. I don't get to solve UPnP problems on the nodes anymore because I don't get to solve any problems on the nodes anymore. The relief and the loss are the same event seen from different angles. Anyone running fleet infrastructure on a protocol they don't control should be thinking about which side of that trade they'd end up on.

Five layers, one migration — what a fabric cutover exposed about storage

John Jacob (JJozzieTech) — Thu, 24 Sep 2026 19:56:08 +0000

The plan was a network change with a storage flavour. Move a Windows workload and two TrueNAS servers off the legacy flat network and onto the new dual-TOR fabric on the workload VLAN. Three cutovers. Repoint the SMB mappings. Done.

The workload moved. Both TrueNAS servers moved. The mappings got repointed. Everything the change plan named came back healthy.

What the plan didn't cover was what "healthy" meant at each layer of the storage stack — because there isn't one layer. There are at least five, they can each be independently green or broken, and a healthy signal at one of them does not imply a healthy signal at the next. That's the piece.

Worth saying up front: I did not know the migration was going to teach me this while it was happening. During the incident, each failure looked like its own problem. It was only after everything was back that the pattern surfaced. Five different failure modes, five different layers of the storage stack, five different validation steps that would have caught each one independently. The layered framing is retrospective. If it reads as if I was tracking it that way in real time, that's an artefact of writing about it, not what the day felt like.

The five layers, in the order they surfaced during the migration:

Network reachability — physical link plus L2/L3 configuration
Service reachability — the SMB advertisement, independent of what's behind it
Filesystem state as the middleware sees it — TrueNAS's cached view of pool status
ZFS state as the kernel sees it — pool importable, disks readable, labels intact
Physical storage-path state — the SAS and CAM layer under ZFS

Each of the five failed at least once during the day. Each of them failed in a way that would have been invisible from the layer above.

// layer 1: network reachability

The Windows workload cutover was the first to surface a network-layer failure. Old address on the legacy fabric, new address on the workload VLAN, TOR-B. Cable in, port configured, physical layer up. The switchport showed a valid 10G link. Windows showed the interface as Disconnected.

That's the layer-1 problem in one sentence. The physical link doesn't prove the operating system has a usable NIC.

The workload host had a 10G adapter installed but no driver loaded for it. The only interface Windows recognised was the onboard Intel I219-LM 1G. From the switch's side everything looked correct. From the workload's side there was no interface to bind an address to. Both views were internally consistent. Both views were incomplete.

The fix was the standard one — install the correct 10G adapter driver, watch the interface enumerate in Windows, watch the switchport carry the traffic that had been waiting for it. The lesson isn't the fix. The lesson is that the switchport LED and the OS network stack are two different signals, and only the OS signal proves the host can actually use the link. On the migration checklist "link up" needs to mean link up on the host's side, not the switch's side.

The second network-layer failure surfaced when I tried to reach one of the TrueNAS servers on its new address. It didn't respond. Ping timed out. RDP timed out. The workload couldn't enumerate its shares because it couldn't reach the server at all.

I don't remember the specific moment that turned into an investigation — I might have gone hunting on the switch first, I might have gone straight to the server. What I found was that the workload VLAN hadn't been added to the LAGG on the TrueNAS server. The switch was configured. The physical link was up. The other end of the LAGG on the server had no idea the workload VLAN existed as a tagged interface.

Adding the workload VLAN as a tagged interface on the existing LAGG, with the server's new address on it, brought the server online. From the workload side, the SMB shares immediately became reachable.

Both of these are "network" failures in the loose sense, and both had the same shape: healthy signal on one side of the link, invisible break on the other. Layer 1 validation isn't "the switch shows the link up." Layer 1 validation is "both endpoints can talk to each other at the address the change plan says they should." Every migration touching L2 or L3 configuration needs to check both directions before declaring the layer green.

// layer 2: service reachability

With the network layer working, the workload tried to enumerate its SMB shares. Most of them appeared. One of them — the largest one, the primary storage target for the workload — returned a Windows error: System error 67 has occurred. The network name cannot be found.

That message is the operating system telling you the SMB server accepted your connection but refused to serve you the share you asked for. It's not a network failure. It's not an authentication failure. It's the SMB service saying yes, I know that share name, and no, I can't give it to you right now.

The dataset backing that share didn't exist.

Specifically: the share was advertised by TrueNAS's SMB configuration, correctly. But the ZFS dataset the share was configured to expose wasn't mounted. It wasn't mounted because the pool holding it wasn't imported. It wasn't imported because the pool's disks weren't visible to the operating system. And the disks weren't visible because the JBOD shelf holding them hadn't been powered on after the re-rack.

The layer here is service reachability — SMB advertising a share is one thing, the share having a valid backing dataset is another thing entirely. TrueNAS doesn't disable an SMB share when its backing dataset goes away. The share stays advertised. The share stays in the configuration. Windows can see it in the list of shares on the server. It just can't be used, and the error message it produces is deliberately unhelpful about why.

Powering the JBOD brought the disks back to visibility. But that only fixed the physical prerequisite. The middleware layer above ZFS still had its own idea of what was going on, and that's the next layer.

// layer 3: middleware view vs ZFS reality

With the JBOD powered, camcontrol devlist on the TrueNAS server showed all the drives. zpool import — the command that lists importable pools — showed the three pools from that shelf as ONLINE and ready to import. From the ZFS layer's perspective, the pools were healthy and available.

The TrueNAS GUI disagreed. Every one of those three pools was still shown as OFFLINE, in the state the middleware had recorded when the JBOD went dark. From the middleware's view, nothing had changed since the shelf went off.

That's the third layer. TrueNAS caches pool state in its middleware database. When something happens outside the middleware's visibility — a JBOD losing power, a SAS cable being pulled, a controller resetting — the middleware doesn't automatically catch up when the situation resolves. The kernel's view of ZFS updates in real time. The middleware's view updates when it's told to.

The recovery for each pool was the same sequence. Export/Disconnect from the GUI, being careful not to check the destroy-data or delete-configuration boxes. Then re-import via the GUI's Import Pool workflow. The pool would enumerate correctly, the middleware would record it as ONLINE with its current disks, and the shares configured against its datasets would start serving. Three pools, three cycles of Export/Import, three shares recovered.

The middleware-vs-ZFS divergence isn't a bug in TrueNAS. It's a consequence of the middleware being a caching layer over the kernel-level truth. The important operator move is knowing that the two views can drift, and knowing which one to trust when they disagree — the answer, always, is the kernel. If zpool import says the pool is available and the GUI says it's offline, the pool is available.

// layer 4: ZFS state under SAS path interruption

The second TrueNAS server surfaced the piece's first real ZFS-layer failure. Several SMB mappings from the workload connected fine but couldn't enumerate files. Timeouts, empty directory listings, hangs on read.

I opened the GUI and looked. One of the pools — call it Pool-A — was in the UNAVAIL state, with all eight of its members marked REMOVED.

That's a scary status line. It suggests the pool has lost every one of its physical members simultaneously. If that were true it would mean the array had suffered a catastrophic multi-drive failure, and the response would be one of the worst calls an operator makes: try to import from backups, accept whatever data loss the last replication window implies, rebuild.

It wasn't that. It looked like that. Because the pool had lost every disk at once, the diagnostic instinct that reaches for "eight independent drives don't fail simultaneously" needed to fire before the instinct that reaches for "restore from backup." A whole pool showing all members REMOVED at once is almost always a path problem, not a media problem. And a path problem is recoverable at the software layer if the labels on the disks are still intact.

I checked. camcontrol devlist showed all eight drives. Their GPTIDs — the ZFS-tracked partition identifiers — had reappeared. zdb -l on each of the drives showed valid ZFS labels naming Pool-A as their pool. SMART data on each of the eight came back clean. Nothing about the disks themselves said they were unhealthy.

The pool's UNAVAIL state was a stale view. zpool clear on Pool-A brought it back ONLINE, all eight members ONLINE, no permanent errors. From the ZFS layer's perspective, the pool was fully recovered.

Except SMB access from the workload was still broken.

The layer-5 lesson to come is that below ZFS is the SAS and CAM path, and that's the layer that had interrupted. The layer-4 lesson is smaller but easy to miss: pool state and consumer state can diverge. Recovering the pool at the ZFS layer doesn't automatically recover the services consuming that pool. Samba, in particular, holds ZFS VFS handles that go stale when the pool goes away and don't refresh when the pool comes back. From Samba's point of view, the dataset it was serving still didn't exist, even though zpool status said it did.

Restarting the SMB service — service samba_server restart from the shell, or Stop/Start on the SMB service in the GUI — dropped the stale handles and rebuilt them against the newly-recovered pool. Windows access came back immediately. Layer 4 clean.

// layer 5: physical storage-path state

The last failure, and the hardest, was on the same TrueNAS server. A second pool — call it Pool-B — was also showing UNAVAIL, also with all eight members REMOVED. The diagnostic sequence I'd just run on Pool-A said the same story: eight members visible in camcontrol devlist, ZFS labels intact, SMART clean. Same recovery expected.

zpool clear on Pool-B did nothing. The pool stayed UNAVAIL. zpool reopen — the command that tells ZFS to re-scan the vdev's devices — did nothing. zpool online on individual members did nothing. None of the ZFS commands could bring the pool back.

At this point I looked at the kernel logs. dmesg was full of messages I hadn't seen on Pool-A. CAM Periph destroyed events. SCSI transport errors. The CAM layer — FreeBSD's Common Access Method, the driver layer that sits between ZFS and the SAS host bus adapter — was in a state where it had torn down its representation of Pool-B's devices and couldn't rebuild them without a full reset.

Then the situation started to spread. Other pools on the same TrueNAS server started showing UNAVAIL in the GUI, one after another. zpool status on any of them returned slowly, then hung. Trying to interact with the affected pools started hanging the shell that ran the commands. The system was becoming unresponsive.

This is a below-ZFS problem. ZFS can only manipulate what CAM can present to it, and CAM can only present what it can reach through the SAS layer. When the SAS/CAM path wedges, ZFS commands become impossible not because there's anything wrong with ZFS but because the layer below it has lost track of what devices exist. There is no ZFS-level command that fixes a stuck CAM layer.

The clean recovery, once I'd worked out what I was looking at, was blunt: stop SMB (to prevent consumer-side hangs during the reset), hard power off TrueNAS1, fully power-cycle both attached JBOD shelves, let everything settle for a couple of minutes, then power the server back on. That resets the entire storage path from the server's HBA outward — the SAS controller re-enumerates the shelves, the shelves re-enumerate their drives, CAM builds a fresh representation from scratch, and ZFS gets a stable substrate to work against again.

After the reset, both TrueNAS servers came back with all their pools ONLINE. Neither the drive labels nor the ZFS state had actually been damaged by any of this. What had failed was the layer under both of them.

The layer-5 lesson is the one that took me longest to internalise: below ZFS is a stack of physical-path components — SAS controller, cables, JBOD backplane, drive interfaces — and when any of those wedge, no amount of ZFS-level work can fix the situation from above. zpool clear cannot repair a hung CAM state. zpool reopen cannot re-enumerate devices the driver layer has torn down. Sometimes the only recovery is a full storage-path reset, from the HBA outward, and that means the server comes down for the duration.

The corollary is a diagnostic one: when zpool commands start hanging on a pool whose disks are physically present and healthy, don't reach for more ZFS-level tools. Look at CAM. Look at the kernel logs. Look for signs that the layer below ZFS has become the problem — because when it has, ZFS is the wrong tool for the job.

// what the migration design did right

The five failures across the day are the bulk of the piece, and worth being clear-eyed about: none of them were caused by the migration design being wrong. Every one was an existing condition or a physical-layer artefact that the fabric change happened to surface.

What the design did right was containment. The workload VLAN on the new fabric isolates this workload's storage traffic from everything else running on the rack. The TOR-A/TOR-B split gives each end of the LAGG its own upstream. The workload servers and the storage servers on the new fabric can talk to each other. Nothing on the legacy network was affected by any of the failures above.

That containment is the reason a bad day for the workload's storage stack was not also a bad day for anything else on the rack. The Kubernetes cluster kept running. The other DePIN workloads kept running. The rack's other services didn't notice the migration was happening beyond the fact that traffic patterns shifted subtly as the workload's dependencies moved. That's not a lucky outcome. It's what per-service VLAN segmentation is for, and it worked as designed.

// what I'd make instinctive

Five doctrine points came out of the day, one per layer.

Layer 1 — network reachability. Switch-side signals and host-side signals are two different things. The switchport LED tells you the link is electrically up; only the host OS's network stack tells you the link is usable. Every migration checklist item that says "verify link" needs to be explicit about which side is being verified. Both sides. Always.

Layer 2 — service reachability. SMB will happily advertise a share whose backing dataset is missing, and the error message clients get is deliberately opaque. Validating an SMB migration means validating that each share can not only be seen but can also be opened and read from — because "visible in the share list" and "backed by a real dataset" are two different states.

Layer 3 — middleware view vs kernel truth. When a management layer caches state over a kernel-level source of truth, the cache can drift. Export/Import in the TrueNAS GUI is the standard way to refresh a pool's middleware view after something upstream has changed. Recognise the situation early — GUI shows OFFLINE, zpool import shows ONLINE — and don't waste time trying to fix the middleware view through the middleware. Cycle it.

Layer 4 — ZFS state under SAS path interruption. A pool showing every member REMOVED simultaneously is almost always a path event, not a media event. Before touching backup recovery or destructive commands, verify the disks with camcontrol devlist, verify the labels with zdb -l, verify SMART. If the disks are healthy and the labels are intact, zpool clear is likely all that's needed. Also — don't forget that pool recovery isn't service recovery. If Samba was serving the pool, it needs to be restarted to drop stale VFS handles.

Layer 5 — physical storage-path state. ZFS can only work on what CAM can present. When CAM wedges, no ZFS command will get through. Kernel logs — specifically CAM Periph destroyed and SCSI transport errors — are the signal that the problem has moved below ZFS. Recovery may require a full physical power-cycle of the storage path from the HBA outward. Plan for that possibility on any migration involving SAS-attached JBODs.

// closing

The rack power migration piece closed on the observation that a change plan can be right and still be incomplete — the plan describes what you know, and the rack describes what actually is, and the gap between them is where changes reveal themselves. This piece is that observation transposed one axis. A layer can be green and the whole still doesn't work, because storage isn't one thing. It's a stack, and each layer of the stack has its own way of failing, its own way of appearing healthy, and its own validation step that has to be run before the layer above it can be trusted.

The migration itself finished the day it was supposed to. Both TrueNAS servers on the new fabric, all pools ONLINE, all shares serving, the workload back in operation on the new fabric. What actually got done that day wasn't just a network cutover — it was a five-layer diagnostic, retrospectively organised into a pattern I could name.

If you're running a storage migration and your validation checklist has one entry per host, you have four fewer entries than you need.

/etc/environment is not read by systemd, and I believed it was for a month

John Jacob (JJozzieTech) — Thu, 17 Sep 2026 20:41:46 +0000

UV_USE_IO_URING=0 was supposed to be the fix. In late August the Flux Cumulus fleet running on Ubuntu 22.04 picked up a nasty failure mode on Node 1: npm install entering uninterruptible sleep, refusing to die, and taking the node from CONFIRMED to expired inside a few hours. The kernel journal named the io_uring submission-queue polling thread:

[Fri Aug 21 08:22:44 2026] INFO: task iou-sqp-1334:1346 blocked for more than 120 seconds.
[Fri Aug 21 08:22:44 2026]       Not tainted 5.15.0-187-generic #197-Ubuntu
[Fri Aug 21 08:22:44 2026] task:iou-sqp-1334    state:D stack:    0 pid: 1346 ppid: 1333 flags:0x00004000
[Fri Aug 21 08:24:45 2026] INFO: task iou-sqp-1334:1346 blocked for more than 241 seconds.

Same task, same PID, still blocked two minutes later. state:D — uninterruptible sleep. SIGKILL does nothing to a task blocked inside the kernel. ppid: 1333 — the parent is the npm install shell.

Node 20's libuv routes async filesystem work through io_uring, and npm install is almost entirely async filesystem work. On this kernel that combination produced the deadlock captured above. UV_USE_IO_URING=0 tells libuv not to use io_uring — the variable is documented in Node's CLI reference, implemented by a dependency, with no stability guarantees attached. Set it before Node starts and the path that hangs never runs. I did not chase the kernel-side root cause: a reproducible trigger and a workaround that eliminated it was enough at the time.

I set it in /etc/environment on every node and verified it was in place.

For most of the next month, it was never in effect at boot on any node.

// the verification that proved the wrong thing

I verified the fix in August by reading the pm2 God Daemon's environment directly out of /proc. pm2 kill && pm2 list to force the daemon to restart, then sudo cat /proc/$(pgrep -f "God Daemon" | head -1)/environ | tr '\0' '\n' | grep UV_USE. The variable was there. The daemon had it.

The check was correct. What it proved was not what I thought.

/etc/environment is read by PAM for login sessions. When I ran pm2 kill && pm2 list in my SSH session, PAM had already given the shell the variable. The daemon pm2 list spawned was a child of that shell — it inherited the environment from its parent. My /proc check then read the child's environment and correctly reported the variable was present.

The daemon inherited from the shell, not from any durable configuration. It was a fact about a process that would not exist after the next reboot. The verification passed honestly, and it proved the wrong thing, because the object under test had been created by the act of testing.

That's the lead beat of this piece: a verification that passes honestly and proves the wrong thing, because the object under test was created by the act of testing. Everything downstream is either mechanism or consequence.

// what actually pulled the trigger

The other thing worth knowing is what triggered Node 1's August incident. The deadlock needed something to cause npm install to run. That something was in the standard FluxNode crontab from Step 7 of the deployment guide: @reboot sleep 120 && pm2 restart flux. A fixed 120-second timer, set to fire into the middle of npm install on every single boot.

Not bad luck. A scheduled collision on a loop, with an unkillable process at the end of it.

The timer wasn't Node 1 specific. Every node in the fleet ran the same line. Why only Node 1 hit the deadlock is plausible inference — it was the only node with a genuine long-running npm install for the timer to land in the middle of during that window — but that's inference, not finding. The cron entry itself was universal.

That cron entry is now removed; a */10 DOS-state monitor covers restart recovery instead.

// why /etc/environment never reached the daemon

At the next reboot after that August verification, systemd resurrected the pm2 daemon. The generated unit's environment block set exactly two variables:

Environment=PATH=/home/flux/.nvm/versions/node/v20.9.0/bin:...
Environment=PM2_HOME=/home/flux/.pm2
ExecStart=/home/flux/.nvm/.../pm2/bin/pm2 resurrect

PATH and PM2_HOME. No UV_USE_IO_URING.

The mechanism: systemd's system manager doesn't read /etc/environment. PAM does, for login sessions. systemd does not. A daemon resurrected by systemd at boot inherits exactly what the unit file declares — which in this case was two variables that did not include the one I was relying on.

That's the kind of thing you either know or you don't. I didn't.

// the fix

The fix is a systemd drop-in. On every node:

sudo mkdir -p /etc/systemd/system/pm2-flux.service.d/
sudo tee /etc/systemd/system/pm2-flux.service.d/override.conf > /dev/null <<'EOF'
[Service]
Environment=UV_USE_IO_URING=0
EOF
sudo systemctl daemon-reload

Drop-ins under /etc/systemd/system/ survive unit regeneration and application updates. The variable is now part of the unit's declared environment, so the daemon inherits it whether it was started by hand or resurrected at boot.

No restart is needed. The drop-in applies at the next boot — which is exactly when the workaround is needed. Applied to all seven nodes on 10–11 September, no node leaving CONFIRMED during the change.

Worth noting the fleet runs Node v20.9.0. From 20.11.1 onward io_uring is opt-in rather than opt-out, so a reader on a current Node likely isn't exposed to this at all — the drop-in is protection for the version we're actually on.

// exposed, never bitten

With the fix in place, I swept the kernel journals across every retained boot on all seven nodes, counting iou-sqp and hung-task events:

for b in $(sudo journalctl -k --list-boots | awk '{print $1}'); do
  printf '%3s  %s\n' "$b" \
    "$(sudo journalctl -k -b "$b" --no-pager 2>/dev/null | \
       grep -c 'iou-sqp\|blocked for more than')"
done

Node 1 showed 67 hung-task reports across five consecutive boots in a three-hour window on 21 August, then zero — including the boot that spanned the entire unprotected month. The rest of the fleet: zero across every retained boot.

The gap cost nothing. But that isn't the framing. The @reboot timer was live on every node for the whole month. What kept the fleet from a repeat wasn't the missing workaround — it was that nothing landed in the middle of an npm install window on any other node in that time. Exposed, never bitten.

The rack power migration piece landed on a rhyming lesson about a different safety measure — an autostart flag disabled for a reason, then quietly missing for weeks after that reason no longer applied. The pattern is the same: nominal protection, silent absence, no cost until an unrelated event exposes both.

// the truncation trap

Immediately after writing the drop-in on the first node, I ran the verification I should have run in August:

systemctl show pm2-flux.service -p Environment

It returned a single line, truncated at terminal width, with a trailing > and the pager sitting at (END). All I could see was PATH. The drop-in had worked; the output couldn't show it.

Environment is emitted as one space-separated line, and any pager that folds long lines will truncate it. The fix is trivial: systemctl show pm2-flux.service -p Environment --no-pager | tr ' ' '\n' | grep UV_USE. Cost of the trap: about a minute of thinking I had to rewrite a drop-in that was already correct.

The instrument lying twice in one investigation: once at a cost of a month, once at a cost of a minute. Both times the tool did exactly what it was asked. Both times the framing of the question was where the mistake sat.

// one more thing from the same three commands

The same systemctl show output also revealed the unit's ExecStart: pm2 resurrect, restoring the process list from ~/.pm2/dump.pm2. Anyone applying the older /etc/environment fix pattern — pm2 kill to force the daemon to inherit the variable — walks into a related trap: kill without saving first, and the resurrect at boot restores an empty process list. The node boots with no application.

pm2 save --force after any kill. One command, one sentence.

The wildcard cert saga — eight days from expiry, three failed attempts, one flag

John Jacob (JJozzieTech) — Thu, 27 Aug 2026 19:55:19 +0000

Cert-expiry monitoring is one of those systems that runs quietly for months and then, suddenly, starts asking for your attention with growing volume. On the Akash provider running out of the Sydney rack, the daily-check script had been raising the same warning for a week — quiet drumbeat, background noise until it wasn't. The wildcard cert covering the provider's public ingress had eight days to live.

Eight days is not a crisis. It's not comfortable either. The comfortable window for renewing a wildcard cert that fronts real tenant workloads is measured in weeks, not days, and I did not want to be doing this work at expiry-minus-hours with tenants noticing. So it moved to the top of the queue on 29 July, expiry set at 6 August.

The first instinct was the wrong one. cert-manager is normally auto-magical. It watches its Certificate resources, notices when they're within the renewal window, kicks off a fresh order against Let's Encrypt, and the operator finds out about it via absence — no notification because nothing broke. When the automation goes quiet, the reflex is to assume it just needs a nudge: restart the pod, re-issue the CertificateRequest, watch the reconciler pick things up again.

That reflex was wrong for reasons the first hour of investigation would surface. But the deeper story is what surfaced after that. Three failed renewal attempts. Root causes distributed across four different components — DNS provider, cert-manager, Cloudflare, acme.sh. Rate-limits triggered by the diagnosis itself. And ultimately a fix that landed as a single flag on an already-working tool, once the problem was framed correctly.

This piece is that story. The 8-day countdown, the three attempts that didn't work, the reframing that ended the incident in about half an hour of active work after days of distributed-bug hunting, and the four lessons that stuck.

// the first hour: what killed the renewal cron

Before any renewal attempt, the first hour went to config review. If cert-manager was supposed to auto-renew and hadn't, the question wasn't "what's wrong with the renewal now" — it was "what's been wrong for weeks that stopped the renewal from happening at all."

The acme.sh account configuration held the answer. Credentials were CF_Key + CF_Email — Cloudflare's Global API Key model. Those credentials had degraded, silently rejected for reasons not fully diagnosed at the time. The renewal cron had been executing on schedule the entire time. It had been failing at authentication the entire time. Neither of us — cron nor operator — had noticed. The immediate fix was rebuilding on scoped API Tokens rather than forensically reconstructing the failure cause — an auth-mode swap that doubled as overdue credential rotation.

Discovery mechanism matters. This wasn't found via log review. acme.sh's cron logs did contain terse authentication errors, but they were the kind of error that reads as normal noise if you're not already suspicious. The smoking gun was the config file itself — one line, CF_Key=..., unambiguous once you know Global Keys are being deprecated.

Auth was the technical trigger. The renewal cron dying was the symptom. The structural problem was that nobody was watching whether the cron actually succeeded. The cron was scheduled. The cron ran. The exit code was ignored. The distinction between "the cron executed" and "the cron did what it was supposed to do" was the gap monitoring should have caught, and didn't.

This is the class of failure that turns a routine credential rotation — normally a fifteen-minute job you do during a Cloudflare account maintenance session — into an 8-day fire drill. The credential rotation itself was trivial. The consequence of the credential rotation having happened silently, weeks earlier, without any monitoring signal that it had happened, is what pushed the deadline.

Cron exit codes without monitoring aren't automation. They're theatre.

// combining fixes: credential rotation plus architecture change

The natural response would have been to rotate to a scoped API Token, restart the renewal cron, and consider the incident closed. That's not what happened. The credential rotation got combined with a larger architecture change — one that had been on the "should probably do this eventually" list for months.

The reasoning was structural. Cloudflare's push away from Global API Keys wasn't arbitrary. Global Keys granted full-account write access with no audit trail — one credential capable of every action across every zone on the account, with no way to scope permissions or track which automation had done what. Scoped API Tokens fixed both problems: zone-limited, purpose-specific, individually revocable.

That same reasoning also applied to the DNS-01 challenge path itself. The existing wildcard cert renewal used acme.sh's default TXT placement — write records to the domain's authoritative zone at Cloudflare, remove them after validation. That mode required Global-API-Key-equivalent write access to the entire zone. A safer architecture would isolate the challenge writes to a dedicated CNAME-delegated location — SiteGround holding the primary nameservers, _acme-challenge CNAMEs delegating just the ACME work to Cloudflare, and API Tokens scoped to only that delegated zone.

Doing credential rotation and DNS architecture change as one combined fix meant less integration risk, less cleanup, and fewer moving parts to reason about afterwards. Two-in-one is usually a false economy — but here, the two changes were driven by the same underlying reasoning, and the fix stack for both landed in the same session.

// first failed attempt: cert-manager's zone-walking bug

With API Tokens in place and the CNAME delegation planned, the first renewal attempt went through cert-manager. And this wasn't a single-vector change. Three vectors were shifting at the same time.

The tool was changing: cert-manager as the new standard for the Kubernetes stack, replacing acme.sh. The CA was changing: Let's Encrypt as the new issuer, replacing the incumbent ZeroSSL. The challenge architecture was changing: CNAME delegation replacing acme.sh's default TXT placement.

Each had its own accumulated "should probably do this eventually" reasoning, and two of them were entangled. Wildcard issuance needs DNS-01 regardless of CA — the ZeroSSL setup was already doing that. What made CNAME delegation a prerequisite rather than a nicety was the credential model: default TXT placement required write access to the whole zone, so scoping the token to anything narrower meant moving the challenge records somewhere that existed only for that purpose.

Whether that was a wise bundling decision is worth naming: it was probably too many changes at once. Multi-vector migrations amplify the diagnostic difficulty when they fail, because you don't immediately know which vector is the problem. That was the lesson not yet learned going into the first attempt.

The attempt failed. But not in a way that produced an obvious error.

The failure signature was subtle. cert-manager wasn't returning an obvious error. The CertificateRequest sat in a Pending state — no failure event, no red status, nothing that would light up a dashboard. cert-manager's Cloudflare provider logs showed it querying zones, matching them, and then attempting to place the TXT record for the ACME challenge at a location. And Let's Encrypt's validator was looking for the TXT record at a different location. The challenge never validated. But nothing about the failure looked like a failure — every component was doing what its logs said it was doing.

It wasn't "this is broken." It was "this looks like it's working but the challenge never validates."

That's the class of failure that eats hours. When every log line reads as healthy activity, the diagnostic instinct — "find the error message and follow it" — has nowhere to go. There is no error message.

The trace came from reading cert-manager's Cloudflare provider source directly, understanding its zone-detection logic, and then cross-referencing that logic against the actual DNS zone contents of the affected domain. cert-manager's provider walks the DNS hierarchy from the record name up toward the root, looking for the "authoritative zone" — the zone where it should write the TXT record. That walk followed a CNAME chain. And the CNAME chain led to .snapshots.

.snapshots is a SiteGround backup-access hostname. It's part of the shared-hosting furniture — installed by the hosting provider for their own backup tooling, invisible to you because you didn't set it up and never think about it. If you're not a SiteGround-hosted domain operator you'd never look at it. If you are, you'd still never look at it, because it has nothing to do with anything you actively manage.

Except now it did. cert-manager's zone-walker followed the .snapshots CNAME, decided that was the authoritative zone for the ACME challenge, and placed the TXT record somewhere Let's Encrypt would never look. Every piece was behaving as documented. The whole still didn't work.

The teaching moment isn't about .snapshots specifically. It's about the class of failure where every component reads healthy and the composition fails silently. That class of failure needs a different diagnostic instinct than error-reading. It needs a diagnostic instinct closer to what network engineers use for packet capture — read the actual traffic between components, don't trust either component's log about what it thinks it's doing.

// second failed attempt: the retry loop became the story

With the .snapshots CNAME identified as the zone-walking trap, the second attempt tried a workaround: manually specify the zone cert-manager should target, bypassing the auto-detection. The attempt failed in a very different way.

cert-manager's default behaviour when a challenge fails is to back off and retry. Backoff intervals lengthen with each retry, but the retries never stop until the CertificateRequest is either manually paused or succeeds. That default is correct for transient failures — network hiccups, brief provider outages, momentary rate-limits. It's harmful for persistent failures on already-loaded infrastructure, where the retry adds pressure without being the sole cause of the problem.

The second attempt ran for about twenty minutes before I noticed it had shifted into a distinct failure mode. cert-manager's default backoff made each retry look reasonable in isolation — a single request every few minutes, exponentially spaced. Then Cloudflare 429 responses started appearing in the cert-manager pod logs — rate-limit rejections.

Here the mechanism deserves care. Cloudflare's global rate limit is 1,200 requests per five-minute window, counted account-wide across dashboard sessions, API tokens, and legacy keys combined. When the limit breaks, everything talking to that account gets blocked for the remainder of the window. cert-manager's backoff-driven retries alone weren't generating anything near 1,200 requests in five minutes — but the retry loop was probably enough additional load, on an account already carrying baseline API traffic from other automation, to push the account over the threshold.

Retries were the last straw, not the sole cause. Exact contribution unknown — request-rate telemetry from the incident window wasn't captured, and I can't reconstruct precisely how much of the budget the retry loop was consuming versus what was already in flight.

That makes the lesson more useful, not less. Account-wide rate-limit budgets mean isolated request rates aren't the whole picture. A retry loop that would be harmless on a quiet account can be the tipping point on a busy one. The operator awareness that matters: not "how many requests does this thing generate" but "how many requests does this thing generate on top of what's already there."

The 429s in the cert-manager logs meant other operations against the same account would have been affected during that window too — that's how account-wide limits work by definition. Whether specific other automation on the account actually noticed depends on whether it was making requests during the limited window; I didn't directly correlate specific other-tool failures during the incident.

The retry loop had become part of the problem.

Not the primary cert renewal being wrong — that was still the thing being diagnosed. But the diagnosis, in the form of cert-manager retrying against a persistently failing challenge, was contributing to a rate-limit condition that affected the whole account. Diagnostic activity was making things worse in ways I couldn't fully see.

The immediate fix was to pause the CertificateRequest — one kubectl annotate to stop cert-manager retrying — and wait for the rate-limit window to clear before touching Cloudflare again. That's the mechanical fix. The operator lesson is broader.

Retry loops without circuit breakers can amplify account-wide rate-limit conditions. cert-manager's default retry behaviour is fine for the failure modes it was designed for. It is not fine for the failure mode where the retry adds pressure to a rate-limit budget that's already tight for reasons outside the tool's visibility.

Circuit breakers — "stop retrying after N consecutive failures of the same type" — exist as a concept in most well-designed retry systems. cert-manager's Cloudflare provider doesn't have one aggressive enough to matter in this situation. Knowing that in advance would have saved twenty minutes of adding pressure to an already-loaded account.

// the pivot that wasn't really a pivot

At this point the shape of the incident changed. cert-manager's Cloudflare provider bug — the zone-walking through .snapshots — wasn't going to be a fix I could apply in-session. It was a bug in the provider code, and correcting it would need an upstream patch, testing, release. That timeline was days minimum, and the cert had single-digit days until expiry.

The path forward was cert-manager off the critical path. Which meant collapsing the three-vector migration down to two.

Framing that as a "pivot" isn't quite right. Two of the three vectors — Let's Encrypt as CA, CNAME delegation as challenge architecture — were staying. The tool vector was reverting. cert-manager was coming off; acme.sh was going back on. But acme.sh wasn't going to be doing what it had been doing before. It had been talking to ZeroSSL with default TXT placement. Now it would talk to Let's Encrypt with CNAME delegation.

Same tool. New work.

The mechanical work was smaller than that reframe implies. acme.sh was already installed. Its cron entry was already running — dead, from the failed authentication weeks earlier, but present. The credentials just needed rotation from Global API Key to API Token, already planned as part of the combined fix. The CA switch was one flag on the renewal command. And the challenge path needed pointing at the right place via --challenge-alias.

Or rather, needed to be pointed at the right place via --challenge-alias. The third obstacle was already waiting.

// third obstacle: the `--challenge-alias` discoverability gap

The third obstacle isn't quite fair to call a "failed attempt." It was more that the tooling had exactly the feature the situation needed, and the documentation didn't foreground it that way.

acme.sh supports DNS Alias Mode via the --challenge-alias flag. When you use CNAME delegation to redirect ACME challenges to a different zone — exactly the architecture the combined fix required — --challenge-alias tells acme.sh where to write the TXT record. Without it, acme.sh writes to the default location. With it, acme.sh writes to the aliased location. The flag is documented. The functionality works cleanly once configured.

But finding it required already knowing that's what you were looking for.

The acme.sh DNS Alias Mode wiki page describes the mechanism — "place TXT records at an aliased location via CNAME delegation." It does not describe the mechanism as "the solution to using base-zone NS delegation with a non-API DNS provider." Someone reading the DNS Alias Mode page top-to-bottom, without already knowing they need CNAME delegation for their specific problem, would come away understanding the mechanism but not connecting it to their situation.

I found it via a specific search for the problem, not by reading through the docs comprehensively. If I hadn't already been sitting with "I need to write TXT records at a location other than the default because SiteGround controls my authoritative nameservers and doesn't expose a DNS API," the wiki page would not have surfaced as relevant.

That's a documentation gap worth naming. It's not the same class of gap as missing documentation or wrong documentation. The docs exist. The docs are correct. The docs describe the mechanism accurately. What's missing is the connection between the mechanism and the class of problem it solves. Discoverability is a distinct axis from correctness, and the acme.sh DNS Alias Mode page is a case where correctness is high and discoverability is low.

Naming that gap publicly is more useful than pretending the answer was always obvious.

// the turning point

The turning point in the incident wasn't a moment of insight. It was a moment of reframing.

After the third obstacle, I sat down with the acme.sh docs again. Second read-through. Different question this time. The first read had asked "how do I do DNS-01 challenge with acme.sh?" — that had produced generic DNS-01 documentation, general-purpose, not obviously connected to the specific architecture I now had in place. The second read asked "what's the flag for the setup I've just configured?" — which is a much smaller, much more specific question.

The problem was framed correctly this time. So the answer was findable.

--challenge-alias surfaced immediately once I knew to look for the flag that told acme.sh "the TXT records go to a different zone than the domain's default authoritative one." That's the whole feature description in one line, and it maps directly to what the CNAME delegation was doing at the DNS layer.

What made this a turning point rather than "another thing to try" wasn't the flag itself. It was the shape of the fix collapsing at that moment.

The cert-manager path had been: diagnose the distributed bug across the zone-walker and the DNS layer, wait for the upstream patch, deploy the patched version, hope no new failures surface. Multi-day, multi-component, uncertain.

The acme.sh path was: add two CNAMEs at SiteGround, add one flag to an already-working cron entry, restart the cron, done. Single-digit hours, two-component, deterministic.

When the plan collapses from "diagnose distributed bug across three components" to "add a flag to an already-working tool," you know before you run it whether it's going to work. That's the anti-heroic version of the story. Not a moment of insight. A moment of reframing that made the answer obvious.

// the fix that landed

The mechanical fix was smaller than the diagnosis suggested it would be.

Two CNAMEs at SiteGround. _acme-challenge for the base domain pointing to a Cloudflare-hosted alias zone. The wildcard-challenge variant pointing to the same alias zone. SiteGround's DNS management supports CNAME creation via their control panel; no API required.

At Cloudflare, a dedicated zone for the alias, with a scoped API Token that had write access only to that zone. Not the domain's actual authoritative zone. Not the full account. Just the ACME challenge zone.

At acme.sh, the credential rotation from Global API Key to API Token in the account configuration. Two additions to the renewal command in the cron: --server letsencrypt to explicitly target Let's Encrypt (rather than acme.sh's ZeroSSL default), and --challenge-alias pointing at the delegated location. That's the whole acme.sh-side change.

At the Kubernetes layer, no change was needed. The renewed certificate would be dropped into the same Kubernetes secret acme.sh had been maintaining all along. The Gateway resource watching that secret would pick up the update automatically. Which is where NGINX Gateway Fabric earned its keep in this incident — but that's the next section.

The renewal ran clean the first time under the new configuration. acme.sh requested the challenge from Let's Encrypt, wrote the TXT record to the alias zone at Cloudflare via the scoped API Token, Let's Encrypt validated against the correct location (because CNAME delegation resolved correctly from Let's Encrypt's side), and the fresh wildcard cert issued. Total active work from "the flag surfaced" to "renewed cert deployed" was about half an hour — most of that spent waiting for SiteGround's NS delegation to propagate.

Two DNS records. Two flags. One credential rotation. And two vectors of the original migration (Let's Encrypt as CA, CNAME delegation as challenge architecture) preserved cleanly — cert-manager came off, but the harder-to-reverse changes stayed.

// NGINX Gateway Fabric added one thing for free

One small win worth naming, because it saved a step during the incident that would have otherwise been manual.

The certificate that acme.sh renews lives in a Kubernetes secret. The ingress layer that serves the certificate to inbound HTTPS traffic — in this stack, NGINX Gateway Fabric — reads that secret and terminates TLS with it.

Standard nginx-ingress-controller requires that when the secret changes, the ingress-controller pods either restart or be signalled to reload their configuration. Some deployments handle this via a sidecar that watches Kubernetes secrets and triggers reloads. Others handle it via periodic reload jobs. Some handle it by not handling it, and letting operators reload manually when they remember. All of those approaches add moving parts.

NGINX Gateway Fabric handles secret updates automatically. When the wildcard cert secret updated with the new acme.sh-issued certificate, the Gateway picked it up without any manual reload, without a sidecar, without a periodic job. TLS termination started using the new cert. No operator action required.

Not the story of this incident. But worth naming as a small win: the ingress layer was one of the few things in this whole stack that didn't need touching during the recovery. When something goes right during an incident, it's worth naming — especially when the same functionality in a sibling tool would have added another manual step under time pressure.

// what I'd make instinctive

Four lessons went into the runbook after this.

Cron exit codes need monitoring, not just cron execution. The renewal cron was scheduled correctly. It ran on schedule for weeks. Cron ran successfully — the shell command executed, cron logged the execution, the cron system considered the job complete. And every single execution was failing at authentication. Cron-ran-successfully monitoring is not the same as job-succeeded monitoring. Any critical automation — cert renewal, backup jobs, replication tasks, anything where "the cron ran" doesn't imply "the intended work happened" — needs both.

"Looks like working" is a distinct failure mode from "broken." cert-manager's zone-walking failure produced no error message, no failed status, no red dashboard indicator. Every log line read as healthy activity. The composition failed silently. That class of failure needs a diagnostic instinct closer to network engineering — read the actual traffic between components, don't trust either component's log about what it thinks it's doing. Error-reading skills don't apply when there are no errors.

Retry loops without circuit breakers can amplify already-loaded conditions. cert-manager's default retry behaviour was correct for transient failures and wrong for persistent ones. Twenty minutes of retrying against an already-loaded Cloudflare account was probably enough additional load to push the account into rate-limit rejection — not the sole cause, but a contribution the tool had no visibility into. Circuit breakers matter not just because they stop wasted work, but because they prevent the retry itself from becoming part of the failure. Know which of your automation has circuit-breaker behaviour and which doesn't.

Discoverability is a distinct axis from correctness. --challenge-alias works. Its documentation exists. What's missing is the connection between the mechanism the docs describe and the class of problem the mechanism solves. That gap turned a fifteen-minute config change into an hour of doc-reading. Documentation authors owe their readers not just correctness but discoverability — the framing that lets a reader find the right feature by describing their problem. Both are needed.

None of these are theoretical. Every one of them was learned by doing the diagnostic in the wrong order the first time.

// closing

Cert-manager saga posts tend to read as heroic sysadmin folklore. Someone stares at cryptic errors for days, has a moment of insight, cracks the case, writes it up as a triumph. The genre is well-established.

Most of them shouldn't be. The plans that collapse to a flag or a config change were never really the plans they looked like during the diagnostic phase. When the fix goes from "three-component distributed bug" to "two DNS records plus a config line," the diagnostic wasn't wrong — the frame was.

Reading this piece back, the whole incident could have ended on day one if the frame had been correct on day one. acme.sh was already in the stack. CNAME delegation was already a documented architecture pattern. --challenge-alias was already a supported flag. The pieces were all there. What wasn't there was the framing that connected them. cert-manager as the migration target was pulling attention toward the wrong tool for the specific problem the incident actually presented.

Article 8's GPU passthrough piece landed a similar-shaped observation about a different domain: the error pointed at the GPU, the culprit was the HBA. Different tools, different failure mode, same operator lesson. When the diagnostic is spending time in the wrong place, the frame is usually the problem — not the diagnostic.

If your renewal cron is running without monitored exit codes, that's the single highest-leverage thing you can fix this week. Not because it prevents this class of failure — but because it turns silent multi-week failures into loud immediate ones. Silent long failures are always worse than loud short ones. Fix the monitoring before you need it.

The error pointed at the GPU. The culprit was the HBA. — a Proxmox passthrough recovery

John Jacob (JJozzieTech) — Thu, 20 Aug 2026 20:03:27 +0000

The plan looked textbook. PCIe passthrough an NVIDIA RTX A4000 from a Proxmox 8 host into an Ubuntu 24.04 VM, giving a Filecoin sealing worker direct GPU access for the compute phases that need it. VT-d on the Xeon. IOMMU enabled at the kernel line. OVMF BIOS on the VM. vfio-pci bound to the device IDs. Machine type set to q35 for PCIe support. Standard sequence, followed by every guide, ordered exactly as those guides ordered it.

Then qm start returned, and the VM did not come up.

Not a kernel panic. Not a host lockup. Not a "device not found" error surfaced to the operator. QEMU exited from the host's perspective and left nothing behind — no running guest, no console output, no obvious pointer at what had failed. The Proxmox web UI showed the VM as stopped, exactly as it had before I hit start.

The errors were there. They were just in a different terminal. dmesg -w on the host, running in a second window, produced a burst of IRQ allocation failures and vfio interrupt remapping errors scrolling past at VM startup, then silence. The host itself was completely fine throughout. Nothing was wrong with the box. Only the VM layer was broken, and it was broken in a way that read more like a misconfigured VM than a hardware conflict — which is exactly why the real cause took three sessions to identify.

The segmentation piece covered the network work that came before this. This piece is one of the workload-layer stories that followed — the GPU passthrough that took two evenings longer than it should have, because the error pointed at the GPU and the culprit was somewhere else entirely.

Two things this piece is not. It is not a step-by-step passthrough guide — there are plenty of those, and they are all fine until they aren't. It is the story of what happens when the checklist runs clean and the passthrough still refuses to work.

// what the GPU was actually there to do

Filecoin sealing — the process of turning raw storage into provable, verifiable data on the network — is not one workload. It's two compute phases with fundamentally different characteristics, and only one of them benefits from a GPU.

The first phase is SDR: Sequential Data Replication. It's designed to be resistant to parallelisation — memory-latency bound, CPU-heavy, and immune to GPU acceleration by design. On this hardware, SDR takes three to four hours per sector regardless of what else is in the box. No GPU speeds it up. That is intentional at the protocol level, not a limitation of the implementation.

The second phase is where the GPU matters. Tree building — specifically the TreeRC computations that produce the Merkle tree structure used in Proof-of-Replication — is GPU-eligible. Without acceleration, TreeRC runs on CPU and adds another two to three hours per sector. With the A4000 handling it, the same work completes in roughly fifteen to twenty minutes.

The framing that gets used in Filecoin marketing — "GPU acceleration makes sealing faster" — is technically true and operationally misleading. The GPU doesn't make sealing fast. It makes the second half of sealing fast. A storage provider without GPU acceleration is bottlenecked on tree computation as badly as on SDR, and the observable sealing throughput is roughly the SDR time plus the CPU tree time, sector by sector.

Where the GPU actually earns its place is in pipeline parallelism at scale. With the CPU pipeline running SDR on one sector while the GPU pipeline runs TreeRC on the previous one, sealing throughput becomes gated by SDR alone. That's the operator-relevant payoff — not making any single sector faster, but decoupling the two phases so they can run against different sectors concurrently. For a small storage provider, that is the difference between sealing two sectors a day and sealing five or six.

That was what the A4000 was there to do. Which is why the VM not coming up was a problem.

// the diagnostic that didn't converge

The investigation stretched across three sessions over about a week, mostly limited by the time I had rather than by the problem itself. Each session ran ninety minutes or so, and each one eliminated one correct-looking explanation without producing the actual cause.

Session one focused on the obvious: was the GPU actually bound to vfio-pci, and was the VM config correct?

lspci -nnk -d 10de:24b0

That output showed Kernel driver in use: vfio-pci — the host had surrendered the card, exactly as intended. No nouveau, no nvidia. The VM's PCI passthrough config referenced the correct BDF for the A4000, machine type was q35, OVMF was enabled, memory ballooning was off. Nothing in the config was wrong. That was session one.

Session two went down the driver blacklist and IOMMU group mapping path. /etc/modprobe.d/vfio.conf had the right device IDs listed. /etc/modprobe.d/blacklist.conf blocked nouveau, nvidia, nvidiafb, and snd_hda_intel. update-initramfs -u -k all had been run, and the host had been rebooted since. All correct.

Then the IOMMU group listing:

find /sys/kernel/iommu_groups/ -type l | sort -V

The A4000 at 03:00.0 appeared in one group with its HDMI audio companion at 03:00.1 — expected. But the same group also contained an IBM SAS2008 HBA. The HBA was managing the host's scratch storage, which meant it couldn't be passed through to the VM. And in an IOMMU group, all devices are the atomic unit of passthrough. You pass through the entire group or none of it.

That was where the pointer to the SAS2008 first appeared. But at that moment, I did not yet understand what shared-group membership specifically causes to fail. Group sharing is a widely documented passthrough concern, and the standard advice ("try the ACS override kernel patch") felt like a hack for a build that would need to run reliably long-term. I closed session two aware the SAS2008 was somehow involved but not yet certain how.

Session three was the crystallisation. Not a eureka moment — a slow narrowing. Once every other explanation was eliminated, the IOMMU group problem was the only candidate left. But the specific mechanism — why exactly a non-passthrough device in the same group breaks the passthrough of the target device — needed one more step of understanding before the physical fix made sense.

// FLR: what makes an IOMMU group problem actually a passthrough blocker

IOMMU group membership alone doesn't break passthrough. The Linux community's shorthand — "everything in the group has to be passed through together" — is correct but incomplete. It describes the safety constraint, not the failure mode. Understanding what actually goes wrong requires knowing what QEMU does at VM startup.

When the VM starts and takes ownership of a passed-through PCI device, QEMU has to reset it — bring it to a known state before the guest OS initialises. The reset mechanism it relies on is Function Level Reset (FLR), a PCIe capability that allows a device to be reset in isolation without affecting anything else on the bus. FLR is the clean, safe reset. It's what QEMU wants.

The A4000 supports FLR. The SAS2008 does not. And here is the mechanism: when QEMU tried to perform FLR on the A4000, it could not do so cleanly because the SAS2008 was in the same IOMMU group and could not itself be reset. The interrupt remapping infrastructure that isolates devices during passthrough could not correctly separate the GPU's IRQ handling from the HBA's, because from the IOMMU's perspective the two devices were treated as a single reset domain. The IRQ allocation failed. The VM didn't start.

The error pointed at the GPU because the GPU was the device QEMU was trying to bring up. The culprit was the HBA because the HBA was the device that couldn't cleanly get out of the way.

FLR support is directly checkable from the host. For any PCI device, the reset_method sysfs file lists the reset mechanisms the kernel considers available:

cat /sys/bus/pci/devices/0000:XX:XX.X/reset_method

For the SAS2008, that file did not list flr among its supported methods. That was the confirmation the mechanism I was reasoning about actually applied to this hardware. Once confirmed, the fix stopped being a question of ACS overrides or kernel patches. It became a question of physical topology.

This is the class of understanding most passthrough guides skip. They tell you what to configure. They don't tell you what to check when the configuration is right and the passthrough still fails. FLR-vs-non-FLR devices sharing an IOMMU group is the silent blocker at the heart of this failure mode.

// moving the HBA to a slot that mattered

The fix was physical. The SAS2008 had to move to a different PCIe slot — one that was not just physically distant from the A4000 but sat behind a genuinely separate PCIe root port. That distinction is critical and easy to miss.

Not every PCIe slot on a server board sits behind its own root port. Slot layout on the motherboard reflects physical trace routing, not IOMMU topology. Two adjacent slots may share a root port; two slots on opposite ends of the board may or may not. Moving the HBA one slot over on the same root complex would have done nothing — the IOMMU group membership would have been identical after reboot, and the failure would have repeated.

The target slot had to be chosen carefully. Two sources helped. The board's manual documents the PCIe topology — which slots connect to which CPU root complex, and how the lanes are allocated across the PCH. The existing IOMMU group listing also carries information: any two devices that had shown up in different groups before must, by definition, be behind different root ports. A slot known to be safe for the HBA was a slot whose currently-installed device (or its adjacent devices) had shown up in a group separate from the A4000's.

The physical work itself was straightforward. Full shutdown of the host — no live PCIe hotswap for this class of card. Chassis open. HBA out of its original slot. SAS cables temporarily disconnected (they had to be re-plugged after the move — a small operational risk if any cable connector had degraded, but nothing did). HBA into the target slot on the other root complex. Chassis closed. Boot.

Confidence going into that boot was moderate, not certain. PCIe slot physical location does not guarantee separate root complex membership — the only real proof is what the IOMMU group listing shows after boot. Moving the card is a testable hypothesis, not a guaranteed fix. If the target slot had turned out to be on the same root complex as the A4000 anyway, I would have been back at square one, chassis open again.

The fix confirmed itself in layers.

Structural first. After boot, the IOMMU group listing showed the SAS2008 in its own group, and the A4000 with only its audio companion at 03:00.1. That was the proof the physical move had produced the isolation it needed to produce. Before I even started the VM, I knew the group problem was resolved at the topology layer.

Functional second. VM started. QEMU didn't exit. Guest OS came up. nvidia-smi inside the VM reported the A4000 with its full 16 GB of memory and no error state. CUDA initialised. The card was talking to its driver.

Ecological third. The sealing worker initialised, picked up a queued sector, and the first GPU-accelerated TreeRC task completed in fifteen minutes instead of three hours. That was the confirmation the whole endeavour had been about.

Three checks, each validating a different layer, each necessary. The structural check proved the topology; the functional check proved the driver stack; the ecological check proved the workload actually benefited. Skipping any one of them would have left the fix incompletely verified.

// the second time was clean

The Filecoin project was later scrapped for reasons unrelated to the GPU passthrough — project constraints outside this story. The A4000 came back into the general hardware pool. When MMX came online later as a farming workload, the passthrough pattern needed to run again on a different guest OS.

MMX is a Chia fork with an important twist relevant to storage layout: it supports both HDD plots and SSD plots, and the SSD plots are compressed. Farming compressed plots requires GPU-accelerated decompression at every proof challenge — the harvester has to expand the plot data on the fly to check whether it holds a winning proof. Without a GPU, compressed SSD plotting is effectively non-viable at scale. With one, the decompression happens fast enough that the harvester keeps up with challenges in real time.

MMX and Chia have always run on Windows VMs on this network — a convention that predates this build, driven purely by the maturity of the Windows GUI tooling around Chia's ecosystem rather than any technical requirement. So the second passthrough build targeted a Windows guest instead of Ubuntu.

The build was clean. IOMMU group listing checked before touching physical hardware — target slot known-safe. Device IDs identified. VFIO binding confirmed. Windows installed with only the emulated VGA present, drivers applied after boot, GPU attached in a second pass to avoid the display blackout risk when the NVIDIA driver takes over primary output. Two extra CPU flags on the VM: hidden=1 to mask the hypervisor and avoid the notorious Error 43, and +pcid for TLB performance under memory-heavy workloads.

One clean build. No gotchas. Roughly one session, no diagnostic sessions after.

The pattern only reads as clean because the checks became routine. That is what the Filecoin project's expensive lesson bought.

// what I'd make instinctive

Four things went into the runbook after this.

Check IOMMU groups before touching physical hardware. The listing takes seconds to produce and tells you exactly what the passthrough constraints will be before you start. Every hour of build effort saved by checking the group listing first is an hour that doesn't need to be spent later, chassis open, moving cards under time pressure.

Non-FLR devices in the same group are silent blockers. IOMMU group membership on its own doesn't fail the passthrough. It's the combination of shared group and one device without FLR support that produces the specific failure mode that took me three sessions to identify. If your target device is in a group with a device that doesn't support FLR, that group needs to be broken up before passthrough will work. The reset_method sysfs check is thirty seconds and catches this class of problem definitively.

Physical slot choice is a topology question, not a proximity question. Adjacent slots may or may not sit behind separate root ports. Distant slots may or may not either. The board manual and the existing IOMMU group listing between them tell you which slots are actually independent. Guessing based on physical distance is worse than guessing at random, because it feels like a plan.

Layered confirmation matters — structural, functional, ecological. A working passthrough is not the same as a working workload. The structural check (IOMMU group listing) proves the topology. The functional check (nvidia-smi or equivalent) proves the driver stack. The ecological check (the actual workload running against real work) proves the whole system does what it was built to do. Skip any layer and the fix is only partly verified.

None of these are theoretical. Every one of them was learned by getting the diagnostic order wrong the first time.

// closing

The GPU passthrough pattern extends far beyond either of the workloads that surfaced in this piece. Any hardware acceleration that a VM needs — an AI or ML lab running CUDA training jobs, a transcoding server that needs hardware video encoders, a gaming VM that needs a discrete GPU, or a media server that needs GPU-accelerated encoding for streaming to remote clients — hits exactly the same setup questions, the same IOMMU group constraints, and the same class of neighbour-check that this piece is built around.

The unified lesson from this piece is small and load-bearing: the error pointed at the GPU. The culprit was the HBA. Check the neighbours before you blame the card.

If you're planning GPU passthrough on a Proxmox host, do this before you touch any physical hardware. Enumerate the IOMMU groups on the current configuration. Identify what shares a group with your target device. Check the reset_method sysfs entry for each of those neighbours. If any of them lacks FLR support, plan the physical move now — not later, when the VM refuses to start and the error message is pointing everywhere except at the actual problem.

The checklist runs clean. Until it doesn't. This is the check that catches when it doesn't.

Recovering a ZFS pool with a missing RAIDZ1 vdev — without wiping a single disk

John Jacob (JJozzieTech) — Thu, 13 Aug 2026 20:25:55 +0000

The same outage that suspended the first pool also took down a second pool on the same box in a different, harder way. That's the piece the suspended-pool recovery closed by pointing at. This is that piece.

The first pool showed multiple disks marked as faulted, but the vdev structure was intact. It was a path issue that looked catastrophic. The second pool, on the same box, hit by the same outage, showed something worse: an entire RAIDZ1 vdev in the UNAVAIL state, with every disk in that vdev marked as REMOVED. The other vdev in the pool was fully ONLINE. But that didn't matter.

A ZFS pool striped across multiple vdevs is only as available as its least-available vdev. If one vdev is unavailable, the pool is unavailable, even if every other vdev is healthy. Pool-level parity does not protect against a whole vdev going away — that's not what ZFS's striping model does. If you want the pool to survive one vdev disappearing, you need mirror-of-vdevs or dRAID or another topology that plans for that specific failure. RAIDZ1 across vdev-striping does not.

I refer to the affected pool as MediaPool-B, matching the naming from the sibling piece.

The recovery for this pool was fundamentally different from the first one's. The first pool's recovery was zpool clear after checking the disks were visible. This pool's recovery required reading the ZFS labels directly to confirm the "removed" disks still had valid pool metadata — before running any command that could touch state. And after the pool came back online, there was a second problem waiting: several hundred files with permanent errors that had to be handled cleanly before the pool was truly clean again.

// what the console showed

The pool's status output, structurally:

MediaPool-B  SUSPENDED
  raidz1-0   UNAVAIL
    disk 1   REMOVED
    disk 2   REMOVED
    disk 3   REMOVED
    disk 4   REMOVED
  raidz1-1   ONLINE
    disk 1   ONLINE
    disk 2   ONLINE
    disk 3   ONLINE
    disk 4   ONLINE
    disk 5   ONLINE
    disk 6   ONLINE

Two RAIDZ1 vdevs in the pool. One fully online. One showing every disk as REMOVED and the vdev itself as UNAVAIL. The pool as a whole in SUSPENDED state.

The visual read of that output is that half the pool has evaporated. The instinct is to look at the TrueNAS GUI, see disks appearing as "unassigned," and reach for either "Add to Pool" or "Wipe." Both are wrong. Both are, in fact, the most dangerous things you can do in this situation.

The reason is a distinction the TrueNAS SCALE GUI doesn't surface clearly. A disk shown as "unassigned" in the GUI is a disk that TrueNAS is not currently associating with a pool through its own tracking. That's not the same as a disk without a ZFS label on it. Every disk that was ever a valid ZFS pool member has labels on it — four of them, redundantly placed at the start and end of the device — describing the pool it belongs to, the vdev it was part of, and the identifier that makes it a member. Those labels survive the pool being suspended. They survive TrueNAS forgetting about the pool. They survive a reboot. What they don't survive is you wiping the disk, or adding the disk to a new pool structure. Both destroy the labels. Neither is recoverable.

So the first correct move is exactly the one the GUI is designed to push you away from: don't touch the disks yet.

// the critical move: don't trust the GUI

The right move is to read the ZFS labels directly, before any GUI action, to prove those "unassigned" disks are still legitimate members of the pool that ZFS thinks is broken. The tool is zdb, ZFS's low-level debugger, and the flag is -l for label inspection.

The pattern I used was a shell loop that scanned every partition on every disk, called zdb -l on each, checked whether the label mentioned the pool by name, and printed the relevant identifying fields for any match:

for d in /dev/sd*2; do
  zdb -l "$d" 2>/dev/null | grep -q "MediaPool-B" && \
    echo "===== $d =====" && \
    zdb -l "$d" 2>/dev/null | egrep "name:|guid:|top_guid|path:"
done

The output came back with entries for the four disks that had been shown as REMOVED. Each entry had a name: field matching MediaPool-B, a guid: uniquely identifying that member disk, and a top_guid: field identifying which vdev the disk belonged to. The four top_guid: values matched each other — meaning the four disks did all belong to the same vdev — and matched the top-vdev GUID of the missing raidz1-0.

That was the turning point of the recovery. The four disks were not gone. They were not damaged. They were not the wrong disks. They were exactly the four members of the missing vdev that they'd always been, still present with their labels intact, waiting for ZFS to resume the pool.

This is the class of check that matters at the moment when the GUI's information is missing the shape of the actual state. zdb -l doesn't care what TrueNAS thinks the disks are for. It reads what's actually written on the disk. That distinction — between what a management layer says about a disk and what the disk itself says about its history — is the whole point of ZFS's label design. Nothing at the storage layer should trust the management layer's model of state over the storage layer's own record.

If any of the four disks had failed to show a matching label, or had shown a label for a different pool, the response would have been different. In that case, the correct move would be to stop, escalate to the physical layer, and only come back to software recovery after understanding why a disk was reporting inconsistently. Nothing was inconsistent. Every disk reported exactly the label expected. The pool was recoverable.

// zpool clear, this time with confirmation

With the four missing disks confirmed as legitimate vdev members via their labels, the recovery command was the same one the first pool's recovery used:

zpool clear MediaPool-B
zpool status -v MediaPool-B

The pool came back to ONLINE. All disks online. READ, WRITE, and CKSUM counters at zero across every device. The vdev that had been UNAVAIL was now healthy. The pool was importable, mountable, and — critically — the labels told ZFS the same story the labels had always told, so the pool's history and identity carried through cleanly.

This is where the confidence delta from the first pool's recovery mattered. In the first pool, zpool clear was applied after confirming the disks were visible and the kernel path was quiet. Here, it was applied after confirming the disks weren't just visible — they were still legitimate members of the pool ZFS was trying to bring back. Different level of certainty, different level of stakes. A blind zpool clear on the first pool's shape would have worked. A blind zpool clear on this pool's shape, without the label check first, would have been a bet on state that hadn't been verified.

But zpool status after the clear showed something the first pool didn't produce.

// the second problem: permanent errors

The pool was ONLINE. Every disk was ONLINE. Every counter was zero. And ZFS was reporting permanent errors in application data — specifically, several hundred files that had errors ZFS couldn't reconcile from parity.

Permanent errors mean exactly what they sound like. ZFS has tried to reconstruct these files from the parity blocks in the vdev, and the reconstruction can't produce the file the pool's metadata says should exist. The file's content is unrecoverable at the ZFS layer. What you do next depends entirely on what the files are.

For this recovery, the affected files were regeneratable workload data — application-layer files that were expensive to reproduce but not irreplaceable. If they'd been irreplaceable — family photos, business documents, source code without a git remote — the correct next step would have been to restore from backup for those specific paths, then clear the error state. Application context determines the correct operational response. There is no single answer.

The sequence for regeneratable data:

Save the evidence first. The zpool status -v output lists every affected file by path, and that list is the ground truth of what's actually corrupt. It should be captured to disk before anything is deleted, so the record survives whatever comes next:

zpool status -v MediaPool-B > /root/MediaPool-B_corrupt_files_$(date +%F_%H%M).txt

Extract just the file paths, filtered to the affected dataset:

zpool status -v MediaPool-B \
  | grep "/mnt/MediaPool-B/Dataset/" \
  > /root/MediaPool-B_corrupt_delete_list.txt

Count and preview before doing anything destructive:

wc -l /root/MediaPool-B_corrupt_delete_list.txt
head -20 /root/MediaPool-B_corrupt_delete_list.txt

In this case, the delete list came out to 311 files, every one of them under the expected dataset path. That confirmed the impact was fully contained to the regeneratable workload before removing anything.

Delete the corrupted files with whitespace trimming — the zpool status output leaves leading spaces on file paths, and rm doesn't handle those well:

while read f; do
  f="$(echo "$f" | xargs)"
  rm -v "$f"
done < /root/MediaPool-B_corrupt_delete_list.txt

Then verify none of the listed paths still exist:

while read f; do
  f="$(echo "$f" | xargs)"
  [ -e "$f" ] && echo "$f"
done < /root/MediaPool-B_corrupt_delete_list.txt | wc -l

Zero. All 311 files gone.

Clear the error state and scrub:

zpool clear MediaPool-B
zpool scrub MediaPool-B

The scrub is what actually retires the stale error header. zpool status immediately after zpool clear will still show the permanent-errors block with no files listed — the header persists until a scrub confirms all data is clean. After scrub completed, the pool status came back to:

MediaPool-B = ONLINE
scrub completed with 0 errors
errors = No known data errors

// what SMART said afterward

Before declaring the recovery complete, I checked SMART health on each of the four disks that had been marked REMOVED. Any of them could have had a real physical issue that contributed to the outage's severity on this pool.

smartctl -a /dev/diskA
smartctl -a /dev/diskB
smartctl -a /dev/diskC
smartctl -a /dev/diskD

Three of the four came back clean — SMART health OK, normal temperatures, zero uncorrected errors, zero reallocated sectors. Fourth disk came back with a small grown-defect count. Not zero, but low enough that immediate replacement wasn't the answer. That disk went on a watch list to be checked at each subsequent scrub, with the plan to pull it if the count grew.

The lesson here is a small one but worth stating: recovery doesn't end at "the pool is back." It ends at "the pool is back and the drives that were involved in the incident have been individually checked for physical health." Neither one substitutes for the other.

// what I'd make instinctive

Four things went into the runbook after this pool.

Unassigned in the GUI is not the same as unused. TrueNAS SCALE shows disks as "unassigned" when its own tracking doesn't currently associate them with a pool. That has almost nothing to do with what's on the disk. A disk that's shown as unassigned can still hold valid ZFS labels for a pool the system is trying to bring back. The GUI's model of state is separate from the storage layer's model of state. When those diverge, trust the storage layer.

Read ZFS labels before doing anything destructive. zdb -l is cheap to run, always available, and reads exactly what's written to the disk. It doesn't rely on any pool being imported, any middleware being healthy, or any GUI showing correct information. When something looks catastrophically wrong with a vdev, zdb -l is the first tool to reach for, not the last. Wiping a disk that still has valid pool labels is destructive and not recoverable — thirty seconds of label inspection prevents that class of mistake entirely.

One unavailable vdev takes the whole pool down. If the pool is striped across multiple vdevs — the default for capacity-oriented ZFS deployments — the pool's availability is the availability of its worst vdev. Mirror-of-vdev or dRAID topologies are the alternatives when one whole vdev going away isn't an acceptable failure mode. Choose the topology deliberately, and know what failure modes you've bought into.

zpool clear doesn't guarantee no permanent errors. The pool coming back online means ZFS has decided the vdev structure is workable. It doesn't mean every file survived. Permanent-error handling is a separate operational step, and the correct response depends entirely on whether the affected files can be regenerated or need to be restored from backup. There is no single answer that's right for every dataset.

None of these are theoretical. Every one of them was learned by doing the work right the first time — or nearly wrong once, which counts for the same lesson.

// closing

Two pools, two recoveries, one outage. The first pool was a path issue that looked like a shelf failure — zpool clear after confirming the disks were visible. The second pool was a vdev issue that looked like half the pool had evaporated — zpool clear after confirming the labels were still valid, then several hundred files deleted, then a scrub. Different failure modes, different recovery sequences, different mental models. Both pools came back without any physical replacement.

The unified lesson across both pieces is small but load-bearing: ZFS's failure reporting is designed to be interpreted, not complied with. The console output tells you what ZFS observed. It doesn't tell you what actually happened, and it doesn't tell you what to do about it. That interpretation is the operator's job, and it depends on understanding what the storage layer is actually asking of you at that moment.

The ongoing storage roadmap for this rack starts with backup posture — the PBS project referenced in the CORE-and-SCALE piece is what turns "recovered without backup" from operator luck into operator design. Recoveries this clean shouldn't have depended on the outage being transient. That's a gap I'm closing, not a story I'm telling as a success.

If you're running mixed-workload TrueNAS today and haven't tested a backup restore recently, that's your homework. It's mine too.

After a TrueNAS ISO reinstall — the two things that broke that no one warned me about

John Jacob (JJozzieTech) — Thu, 06 Aug 2026 19:57:14 +0000

The TrueNAS SCALE machine that runs my SMB shares was in trouble. Windows clients had stopped seeing the shares reliably — intermittent at first, then completely. When I dropped to the shell to check, systemctl status smb returned something worse than a failed service: it returned a service that no longer existed. Same for nmb. Same for winbind. The service wrappers were gone.

Samba's binaries were still there. The actual smbd and nmbd executables under /usr/sbin were untouched. But the systemd units that normally wrap them, and the TrueNAS middleware that manages both, had lost too much of themselves to be repaired by unit regeneration or middleware restart. Something at the service-layer wiring had broken beyond in-place fix.

The recovery answer, once I stopped trying to patch it, was a fresh ISO reinstall and a pool reimport. That got the service layer back. What it didn't do — what nobody had warned me it wouldn't do — was fix Windows authentication or fix ACL permissions on the imported datasets. Two things I'd inherited from the reinstall broke SMB access in ways that had nothing to do with SMB itself.

This piece is the story of those two things. The SID mismatch that made Windows reject correct credentials. The missing default ACL entries that blocked recursive permission repair. And the fix sequence that restored SMB access to what it had been before the service layer disappeared.

Same box as the ZFS suspended pool piece, different failure mode. When the storage layer goes wrong, ZFS suspends the pool and tells you exactly that. When the service layer goes wrong, the layer just disappears and Windows quietly stops answering.

// why the fresh ISO was the answer

The service layer on TrueNAS SCALE isn't just Samba. It's the systemd units that manage Samba. The middleware plugins that TrueNAS uses to expose SMB configuration through its GUI. The internal API endpoints that the middleware calls when you edit a share in the interface. All three layers were compromised.

I tried the standard recovery moves first. Regenerating service units through the middleware. Reloading systemd. Restarting middlewared to force the internal API to reinitialise. None of it took. The problem wasn't a single missing piece — it was structural damage across the middleware, systemd, and Samba integration layers that had drifted enough that regeneration couldn't reconstruct working state from what remained.

That's the moment where the answer stops being surgery and starts being a reinstall. Not the emotional decision — the reasonable one. When a service layer's structural integrity is gone, patching pieces individually just produces a service layer that limps. A fresh ISO reinstalls all three layers cleanly, at their intended versions, in their intended configuration. Fifteen minutes of install, a pool reimport, and the service layer is back to what it should be.

That's what happened. Fresh ISO. Reimport of pools. smb.service, nmb.service, winbind.service all back. midclt smb.status returning cleanly. The GUI managing SMB again without errors.

Then the actual story of this piece started.

// the two things that break after reinstall

With the service layer working again, Windows still couldn't authenticate to the shares. Correct credentials, rejected by every share on every dataset. The problem wasn't SMB anymore.

The SID mismatch.

When TrueNAS SCALE reinstalls, every user account has to be recreated. Same usernames, same passwords, same group memberships — nothing prevents you from recreating them exactly. But Windows doesn't care about your username. What Windows cares about is the SID.

A SID is the Security Identifier that Windows uses to actually attach permissions to identities. Usernames are human-readable labels layered on top; the identifier that gets written into ACL entries is the SID. When TrueNAS creates a user, it generates a SID. When TrueNAS is reinstalled and the same user is recreated, a new SID gets generated. Not the same one. The SID is not derived from the username.

Existing ACL entries on the datasets — the ones from before the reinstall — reference the old SID. Windows looks at those ACL entries, matches its authentication attempt against the SID recorded there, sees no match against the new user's new SID, and rejects the authentication with "Credentials do not match."

The credentials do match. The SIDs don't. Windows is telling the truth in a way that doesn't help you fix it.

The missing default ACL entries.

The second problem shows up when you try to fix the first one by applying corrected permissions recursively. TrueNAS SCALE's ACL editor supports recursive apply — set permissions on a parent dataset, tell it to apply down through everything below, and it walks the tree. That's the normal repair for permission drift.

Recursive apply on imported datasets errors out immediately:

Error: dacl -- Default ACL entries are required in order to apply ACL recursively

That message is opaque unless you know what a default ACL entry is. Access ACL entries govern the files and directories that exist right now. Default ACL entries govern what permissions get inherited by new files and directories created inside a parent. On a healthy TrueNAS dataset both are set. On a dataset imported from a pool after reinstall, only the access entries survived the import — the default entries didn't come across.

Without default entries, recursive apply doesn't know what to inherit. It refuses to write any of the changes rather than apply them incompletely.

// the fix sequence

Once both problems were understood, the recovery ran in a specific order.

Step one: establish default ACL entries.

Recursive apply had to work before anything else was possible. Establishing default ACL entries on the affected datasets was the unblock.

TrueNAS SCALE ships preset ACL configurations that include both access and default entries. The RESTRICTED preset is the conservative choice — root and the dataset owner get full control, everything else is denied by default. Applying it to the dataset writes both access entries and default entries in the correct form, then recursive apply works.

Step two: recreate the SMB user with the intended group memberships.

The SMB user recreation was straightforward through the TrueNAS GUI. Same username, same password, same primary group. The new user got its new SID assigned automatically. Nothing about this step required anything unusual — the account creation is standard TrueNAS SCALE user management.

The systemd and middleware refresh sequence run after the reinstall to confirm the service layer was clean:

midclt call service.generate_all_units
systemctl daemon-reload
systemctl restart middlewared

Those weren't necessary for the ACL fix — the fresh ISO had already given us a clean service layer — but they're the standard TrueNAS SCALE service-layer refresh sequence, and running them confirmed no residual issues before the ACL work started.

Step three: apply corrected permissions recursively.

With default ACL entries in place from step one, and the SMB user existing with a new SID, the corrected ACL now needed to be written down through the dataset tree. The RESTRICTED preset had already established the ACL structure; adding the SMB user with full control on the dataset and applying recursively wrote the correct SID into every ACL entry through the tree.

Step four: verify from Windows.

Windows client connected to a share, authenticated with the SMB user's credentials, got in cleanly. No more "Credentials do not match." The credentials the user was typing had always been correct — they now matched an ACL entry whose SID pointed at the user actually trying to authenticate.

The ACL work took the same session as the reinstall itself. Most of the time was working through the preset and recursive apply through several affected datasets. The recursive apply itself was fast once it stopped erroring.

// what I'd make instinctive

Three things went into the runbook after this.

Snapshot the system config before major changes. TrueNAS SCALE lets you export the entire system config to a downloadable file — settings, users, ACL definitions, share configurations. Before this incident, I hadn't done it. If I'd had a config snapshot from before the SMB service layer failed, the fresh ISO reinstall would have restored not just the service layer but also the user accounts with their original SIDs. The SID mismatch problem wouldn't have existed. That single missing habit was the whole reason step two of the fix took the form it did. Config snapshots now go with every major change to the box.

Pool imports don't recreate everything the datasets need. Default ACL entries are the visible case; they're the one that surfaces an error message. But the principle is broader. When you import a pool into a fresh TrueNAS installation, you get the pool structure and the actual file data — you don't automatically get all the metadata state that the original installation had built up around the datasets. Some of that state has to be rebuilt manually. Assume it, don't be surprised by it.

Know the SID story before you need it. SIDs are how Windows actually attaches identities to permissions, and username is a display convention. Reinstalls change SIDs. Any ACL entry pointing at an old SID will silently fail to match a new user of the same name. The "Credentials do not match" error is misleading because the credentials aren't the problem. The identifier attached to the credentials is.

None of these are theoretical. Every one of them was learned by doing the work slightly wrong the first time.

// closing

This piece is about the layer between storage and client — where user identity meets file permissions, where authentication decisions happen. When that layer breaks, the storage is still fine, the network is still fine, and the client is confused. The SMB failure category is different from the ZFS failure category. Different response, different recovery, different mental model.

The next piece in this series is another storage-layer story — a ZFS pool where an entire RAIDZ1 vdev appeared to be gone, and the recovery required reading ZFS labels directly to confirm the disks were still valid pool members before touching anything. Different layer, different failure mode, different set of things that turned out not to be as broken as they looked.

If you're running TrueNAS SCALE and haven't exported a system config recently, do it now. That's the single highest-leverage thing you can take from this piece. The config snapshot is what turns a fresh ISO reinstall from a story about SID mismatches into a story about ten minutes of downtime.

A Kava validator on my existing stack — how the plan came together, and what the second chain will do differently

John Jacob (JJozzieTech) — Thu, 30 Jul 2026 20:36:36 +0000

The plan didn't start at Kava. It started at the Cosmos Hub, because the flagship chain is where you go if you're not sure where to go.

The Cosmos Hub's active set — the top validators eligible for block rewards — has a floor. To enter it, a validator has to hold more staked ATOM than the smallest active validator does. In mid-2026 that floor sits at tens of thousands of ATOM — hundreds of thousands of dollars at current prices. That's an infrastructure business, not a homelab experiment. It's the size of stake where you're raising capital before you're writing a .toml file.

I don't have that capital, and I wasn't going to raise it to prove a point about self-stake. So the plan reshaped itself. Instead of validating on the flagship, target a smaller Cosmos appchain — one where a modest self-stake actually gets into the active set, but the chain is real enough that the validator work isn't performative. Kava won selection as chain #1 on that criterion. Real DeFi product with actual users, Binance-listed, healthy market cap, an active set that was reachable for a modest self-stake at the time I entered. Enough consequence to make the work matter; enough reachability to make the work possible.

Every validator writeup I've ever read assumes you got past this decision on money and skips to the software. This one starts here because for most people evaluating whether to run a validator, this is where the plan actually ends. I want the piece where it doesn't.

One thing the software will not forgive: the same cryptographic key that earns the validator's rewards will, if it ever double-signs a block, get the operator permanently and irrecoverably slashed. Not a warning. Not a temporary jail. The stake and any bonded delegator stake gets burned by the chain itself. Every design decision in what follows is downstream of that constraint.

// the hardware happened to have the right shape

The T430 wasn't acquired for this. It was already in the rack, sitting mostly idle, repurposed from a Filecoin project that didn't pan out. 512 GB of ECC on the box total, most of it committed to what had been the Filecoin VM, and the validator was going to draw from the balance. Because the RAM was there, the sizing wasn't a scarcity negotiation. There was room to be comfortable rather than minimal.

The validator VM got 8 vCPU and 48 GiB. Kava's documented minimums are lower than that. The provision deliberately sat above the documented floor rather than at it, because the downside of under-provisioning a signing node is asymmetric. A resource-starved node misses blocks. Missed blocks eventually jail the validator. Jailed validators lose bonded stake. There is no upside to a lean signing node worth the risk of a starved one, so it got comfortable margin.

The 48 GiB is a fixed allocation. Balloon disabled. Deliberate: the validator's RAM is reserved, not dynamically reclaimable, so memory contention with any other VM on the same host can't starve it. That decision followed the same pattern the Filecoin VM used, which had proven itself.

Storage: a 3.5 TB SSD, passed through to the VM as a dedicated device, mounted at /data/cosmos. Not carved from the shared LVM pool that holds VM OS disks. That separation was an explicit design choice — OS-disk failure and chain-data failure are two independent things that shouldn't share a failure mode. If the OS disk dies, the chain data survives. If the chain data corrupts, the OS is untouched. Two failures stay two failures.

The IOPS discipline was verify-before-trust. Before the node was trusted with signing, the disk was benchmarked: fsync p99 around 734 microseconds, roughly 5,353 IOPS on the workload that mattered. Adequate for the validator's write pattern, confirmed by measurement rather than assumed by spec-sheet reading. The chain is currently running lean on that volume — around 48 GB used of 3.5 TB available — so the storage layer is under no near-term pressure.

That's the hardware. Nothing about it was a targeted purchase for validating. The box had the right shape because the rack had the right box, and the decisions that made the box right for the previous project turned out to make it right for this one too.

// the snapshot that wouldn't replay

The first real war story landed at the snapshot restore step, before signing had ever happened.

Full sync from genesis wasn't the option — the chain history is large enough that starting from block zero would have run the node for days before it caught up. Snapshot restore is the standard alternative: download a recent state snapshot from a public provider, extract it to the chain data directory, start the node from there. Fifteen minutes of work, then the node picks up from a recent height and catches to tip.

The first snapshot source I tried, Polkachu, produced state that wouldn't replay cleanly. The node started, entered replay, and choked partway through with IAVL state that wouldn't reconcile under standard pruning parameters. Three attempts. Each one an evening. Each one ending the same way.

The fix was switching to a different snapshot source — NodeStake — and the replay went clean the first time. Nothing wrong with Polkachu as a provider; the specific snapshot that day just had state my node couldn't reconcile.

What came out of the pain became doctrine: verify the canonical AppHash from a public RPC before committing to a long snapshot mount. Fifteen seconds of check. If the snapshot's AppHash doesn't match what the chain reports at that height, the snapshot is bad — save yourself the hours. That check now runs before any snapshot restore on any chain in the pipeline. It's a five-line addition to a preflight script that would have saved me three evenings on this build.

// Horcrux 2-of-3: the signing architecture

Threshold signing is where the double-sign constraint gets solved.

Horcrux takes the validator's private key and splits it mathematically into shares. Each share, on its own, is useless — it can't sign anything. To produce a valid signature, a configurable threshold of shares must cooperate. The private key never exists as a whole thing on any machine; it only exists as an assembled signature at the moment it signs. There is no file on any host that, if stolen, gives an attacker a working key.

The topology choice is 2-of-3: three cosigners, any two of which are enough to sign. That specific shape is the homelab-scale sweet spot. It buys two properties simultaneously.

Fault tolerance. Any one cosigner can go down — for maintenance, a crash, a host failure — and signing continues on the remaining two. No single machine is a single point of failure for the whole validator.

No single point of compromise. No one machine ever holds a usable key. An attacker who takes one cosigner gets one share, which is worthless without a second. The double-sign threat is neutralised at the architectural level.

Alternative topologies each fail one of those tests. 2-of-2 has no fault tolerance — lose one, signing stops. 3-of-5 buys more redundancy than a homelab cluster with a handful of hosts needs while widening the surface area an attacker has to work through. For a small cluster with a no-slashing-tolerance constraint, 2-of-3 is the minimum topology that gives both properties. That's the honest reason it's the standard homelab-scale answer.

Physical distribution matters as much as the math. The three cosigners run on pro1, pro2, and pro4. The validator's full node runs on pro5. The standby node runs on pro3. No host runs both a cosigner share and a signing node. If any one physical host dies, you lose at most one share — not one share and the node together. Correlated failures at the host level get decorrelated by placement.

The honest nuance: whether every original placement decision was made purely for that reason or partly reflected which hosts had capacity, I can't fully reconstruct. What I can say is that the principle is deliberate — cosigner hosts should be separate from node hosts — and the same principle got applied more explicitly when the second chain (Akash) came online later. It reads as doctrine, and it functions as doctrine, even if the Kava-era placement was principle plus fit rather than principle alone.

The Kava-specific setup went cleanly. The shares distributed on the first attempt, the cosigners came up in the intended order, and the Horcrux Raft cluster formed quorum without incident. The hard-won signing lessons — the startup-ordering rule, the empty-placeholder-key-file gotcha, the strict host-separation rule — surfaced later on the Akash build. Those live in a future piece, not this one.

// the last check before bonding

Before the validator was ever bonded to the network, the check that mattered most was the one that proved Horcrux was actually doing the signing.

The mechanics: Kava's validator identity is normally represented by a file called priv_validator_key.json on the full node. It contains the raw private key the node uses to sign blocks. When Horcrux is doing the signing instead, that file must not be present on the node — otherwise the node would sign locally with its own key, which defeats the entire threshold-signing architecture and reintroduces the single-key-compromise risk.

The discipline is this: archive priv_validator_key.json before going live. Confirm the node is signing through Horcrux with no local key present. Only then bond stake to the validator. Only after signing-under-bond is empirically confirmed — real signed blocks recorded on the chain, produced by the threshold-signing cluster with no local key on the node — is the archived key destroyed.

The reason for that specific ordering is asymmetric downside again. If Horcrux fails silently and the node is still capable of falling back to a local key, the double-sign protection is theoretical rather than real, and the operator won't find out until it matters. If the local key is confirmed absent and the node signs anyway, the protection is real. If the local key is confirmed absent and the node stops signing, that's a fast, safe, revertible failure — you archive-restore the key, diagnose Horcrux, try again. Nobody's stake is at risk during the check.

That check is what "the double-sign will get you permanently slashed" reduces to, operationally. Everything before it is preparation. Everything after it is delegator-visible.

// first signed block, first delegator

The moment the validator was bonded and started signing, I didn't notice.

That's how it's supposed to go. When you've verified everything before going live — the storage, the sizing, the snapshot, the threshold-signing, the local-key discipline — the moment of going live is a non-event. The chain assigns a block. The validator signs it. Then another. Then another. There is no cinematic moment. There is a voting_power field on the node's /status endpoint that reads as a nonzero number instead of zero, and there are heights ticking upward in a monitoring panel, and the operator does not notice because nothing has broken.

That's success. Cinematic first-signed-block moments belong to validators that went live without checking. The absence of drama is what the pre-flight work was for.

The emotional beat, when it did arrive, was three days later. A stranger — someone I had no connection to, no relationship with, no reason to expect — looked at the validator, made an evaluation, and delegated stake to it. First independent delegator. Two total on the validator now, mine plus this one. That's the moment the whole thing stopped being an experiment and became a service someone was using.

Here's the current state, as of writing:

D+28 operational. Uptime 100%. Window block miss: 0 out of 10,000. Two delegators. That's not brag material — it's the operational state, the same way zpool status in the ZFS recovery piece was the state. Numbers on a public chain, verifiable by anyone. The point isn't the numbers. It's that this is what the pre-flight work produces.

// building the safety net: standby + drilled failover

The single node was the launch state. The current state is different.

There's now a standby node on pro3 — a second full node, kept synced to chain tip, ready to take over signing if the primary goes down. Same chain data, same signing configuration, same Horcrux endpoints. The switch from primary to standby is scripted — one script triggered by the operator with a confirmation prompt, another script to fail back to primary once it's healthy again.

The manual failover benchmark, from an earlier phase before automation, was 12 minutes 55 seconds from primary-down to signing-resumed. Human-driven. Every step verified by hand. Correct, but slow.

The scripted failover, drilled against a real deliberately-triggered outage — not a simulation, an actual systemctl stop kava on the primary — clocked at 1 minute 49.5 seconds from primary-down to signing-resumed. 37 blocks missed against the chain's 10,000-block jailing window. Roughly 0.4% of the jailing buffer consumed for a real failover event. Big enough to matter; small enough to survive.

The interesting part of the drill wasn't the failover script — it was the failback. The failback script's first dry run — the one designed to bring primary back into service after standby had taken over — surfaced two real bugs that had never been noticed. A missing privval-blank step, which would have hung real incidents to the 30-minute ceiling every time. A grep -c || echo 0 double-print anti-pattern that produced false WARNING messages downstream. Both were root-caused, fixed, and re-drilled to a clean end-to-end pass before the script was trusted with real operations. That work is exactly what a drill is for — finding bugs in the safety net before the safety net is what stands between you and being jailed.

One additional operational lesson surfaced during the drill: Horcrux's own systemctl ActiveState can report active before its Raft cluster has actually reformed quorum well enough to co-sign. Same category of "healthy-looking status ≠ real capability" issue as the chain's own voting_power field can produce, now confirmed to extend to the signing layer itself. The failback script now waits for genuine signed events, not for ActiveState=active, before declaring signing resumed.

The full topology and drill results live on the validator page. That page has the numbers; this piece has the shape.

// the first governance vote

Three weeks in, the first governance vote came up. Kava proposal 220 — recover the IBC light client for the Kava-Secret channel. Vote was YES.

The operational shape of casting a vote as a validator is different from casting one as a delegator. As a delegator, you click through a wallet UI. As a validator, you sign a governance transaction from the operator account — not the signing key, the operator key — using a discipline the validator page documents as "brief-import." The operator key is not resident on any always-on machine. It gets imported briefly for the vote, the vote transaction is signed and broadcast, and the key is removed again. Same principle as the signing-key discipline, applied to governance: the sensitive credential is never around when it doesn't need to be.

The substantive reasoning on the vote — why yes, what the recurrence of this fix pattern surfaces about relayer-set health on the Kava-Secret channel, why the meta-governance abstain vote is defensible even if it's not what I voted — lives on the on-site governance record. This piece is about the operational experience of casting the vote, not the merits of the proposal. That's a genuine split of concerns: the validator page and the governance record are the accountability artefacts, this piece is the story.

Cross-link to the governance record for the substance: /validators/kava/governance/220/.

// what would have been day-one doctrine

Some of the war stories in this piece became doctrine that got applied to the second chain. If I were starting Kava again from scratch, this is what would be day-one design rather than lesson-learned:

Startup ordering. Horcrux cosigners must be up and connected before the validator node starts its signing daemon. There's a roughly 3-second privval handshake timeout that will crash-loop the daemon if it opens its listener before a cosigner is ready to respond. Every operational script now has cosigner-side up before node-side up as a hard sequence.

The empty-placeholder-key gotcha. Some chain forks panic on startup if the priv_validator_key.json file is present but empty, rather than if it's absent entirely. The safe posture is to remove the file, not zero it. Deleted, not blanked.

AppHash verification before snapshot mount. The Polkachu-snapshot lesson from earlier. Verify the canonical AppHash from a public RPC before starting the long extraction. Fifteen seconds of check, potentially days of saved rework.

Cosigner and node physical separation as strict rule. Not "we should keep them separate where possible" — a hard rule that a cosigner share must never live on the same host as a signing node. Applied to Akash without exception. Would have been applied to Kava the same way if I'd had the discipline at day one.

Drilled failover, not planned failover. A failover script that has never been run against a real outage is theatre. HA4.5's discovery of two real bugs in a script that looked correct is the general case, not the exception. Drill the failover — with a real systemctl stop, not a simulation — before trusting it with a real incident.

None of these are theoretical. Every one of them was learned by doing the work slightly wrong the first time, and none of them cost real slashing because the safety architecture caught the mistakes before they compounded. That's what the safety architecture is for. But knowing what would have been day-one doctrine is the compounding gain — the second chain benefits from every one of these lessons without having to re-learn them.

// closing

Kava is chain #1. Twenty-eight days operational, 100% uptime, two delegators, one governance vote cast. The build wasn't dramatic. The delegator was.

The next chain is Akash. This time the shares distribution is day-one doctrine, not lessons learned later. The startup ordering is day-one doctrine, not a crash loop caught in staging. The snapshot AppHash check runs before the extraction, not after the third failed replay. The failover script gets drilled against a real outage before the validator ever bonds, not weeks after.

The homelab-first framing lands here: the validator playbook doesn't come from cloud infrastructure or a dedicated hosting provider. It comes from the rack, running alongside the rest of the workloads, doing the drilled operational work that most delegators trust cloud providers to handle. Whether that's the right posture for delegators is a conversation for someone else's article. Whether it's a defensible posture for an operator — one where the same person who wrote the failover script has run it, on real hardware, against a real outage — is what the whole series is trying to demonstrate.

Chain #1 signed its first block without ceremony. Chain #2 will sign its first block knowing what the first chain took a month to teach.

DEV Community: John Jacob (JJozzieTech)

Why I tested from inside the network

// what a speed claim actually measures

// the path I measured

// nine runs across three networks

// the earlier 936 Mbps result

// latency belongs next to throughput

// what the result proves — and what it does not

// the engineering takeaway

// method and evidence note

Six months operating an Akash provider

// what this covers

// the deadline

// what the deadline exposed

// foundation

// the hardware interrupt

// measuring the failover

// six minor versions in a week

// sixty-seven days

// hardening

// a silent two weeks

// the rebuild

// four days of nothing

// unaudited

// the pattern

// the receipts

iSCSI gives you two minutes

// the setup

// act 1 — the false alarm

// act 2 — the real one

// the two numbers I should have known

// the recovery

// what I'd make instinctive

// closing

The storage failure that was actually an ARP race

// why storage was the first suspect

// the Windows ARP red herring

// tracing from client to the fabric

// the ARP table discrepancy that unlocked everything

// the endpoint reveal

// what the duplicate IP actually did

// the fix

// what I'd make instinctive

// closing

Four lines of patched JavaScript, and the router VM that deleted them

// why the patches existed

// the maintenance tax

// why it kept breaking anyway

// the alternative

// the coda that closed everything else

// what I'd make instinctive

Five layers, one migration — what a fabric cutover exposed about storage

// layer 1: network reachability

// layer 2: service reachability

// layer 3: middleware view vs ZFS reality

// layer 4: ZFS state under SAS path interruption

// layer 5: physical storage-path state

// what the migration design did right

// what I'd make instinctive

// closing

/etc/environment is not read by systemd, and I believed it was for a month

// the verification that proved the wrong thing

// what actually pulled the trigger

// why /etc/environment never reached the daemon

// the fix

// exposed, never bitten

// the truncation trap

// one more thing from the same three commands

The wildcard cert saga — eight days from expiry, three failed attempts, one flag

// the first hour: what killed the renewal cron

// combining fixes: credential rotation plus architecture change

// first failed attempt: cert-manager's zone-walking bug

// second failed attempt: the retry loop became the story

// the pivot that wasn't really a pivot

// third obstacle: the --challenge-alias discoverability gap

// the turning point

// the fix that landed

// NGINX Gateway Fabric added one thing for free

// what I'd make instinctive

// closing

// third obstacle: the `--challenge-alias` discoverability gap