In May the provider missed a mandatory upgrade deadline, lost its audit attestation, and watched roughly four-fifths of its active leases disappear inside two weeks.
The recovery took twenty-six days. Most of that time was not spent on the upgrade. It was spent on things that had been quietly broken for months — including one bug that had been suppressing all storage reporting for sixty-seven days without producing a single log line.
That turned out to be the pattern. Every expensive incident in the six months since has had the same shape: nothing failed loudly. Something simply stopped happening, and nothing said so.
// what this covers
May to October 2026, on a production Akash provider running in a Sydney enterprise lab. The provider is older than that, but the documented record starts in May, and I would rather write six months I can evidence than twelve I can't.
Specific numbers where I have them. No numbers where I don't. Nothing here is reconstructed from memory — it comes from after-action reviews written at the time, a daily health check that has been running throughout, and the chain.
// the deadline
The provider ran a mandatory upgrade deadline on 1 May and missed it. Audit attestation was removed. Lease count began falling immediately and was still falling two weeks later.
The first recovery attempt lasted about forty minutes. It stopped at the ingress chart install with a version constraint:
chart requires kubeVersion: >= 1.31.0-0
which is incompatible with Kubernetes v1.28.15
The new ingress controller needed Kubernetes 1.31 or later. The cluster was on 1.28.15, which was already end-of-life. There was no flag to pass, no version to pin back to, no way through.
Recovering the audit meant upgrading Kubernetes first. Which meant the cluster had to survive six minor-version jumps. Which meant it needed a control plane that could lose a node, and backups that existed.
None of those were true.
// what the deadline exposed
The honest inventory, severity-ordered, as it stood when the recovery started:
No etcd backups. None. A single disk failure on the control plane's data directory would have meant total cluster state loss. This had been true for sixty-seven days.
One control plane. Any failure of that one node took the whole cluster down. Also true for sixty-seven days and counting.
Kubernetes end-of-life. No security patches, and the hard blocker on the ingress controller that stopped the first attempt.
Storage reporting empty. The provider could not bid on any order requiring persistent storage, because the inventory operator reported no storage capacity at all. Known for sixty-seven days, investigated repeatedly, unresolved.
The deadline did not create any of this. It made it unavoidable. That is roughly the only good thing to say about missing a deadline: it converts deferred work into work.
// foundation
Backups first, because everything after it was going to be destructive.
An etcd snapshot script on a six-hour cron, writing locally and copying off-host, with seven-day retention. Then — and this is the part people skip — a restore tested end to end into a scratch directory, because an untested backup is a belief rather than a backup.
Then two new control-plane VMs, prepped on two separate physical hosts. Not two total: three. A two-member etcd cluster tolerates zero failures and introduces split-brain risk on a network partition, because either member can believe it holds quorum when it can't see the other. Two members is strictly worse than one. The decision that followed from that — join both new control planes in a single session rather than splitting across two mornings — is the one I would most want another operator to take away from this.
The backups paid for themselves within forty-eight hours, not as a recovery but as permission: every subsequent risky step had a fresh snapshot behind it and therefore did not need to be agonised over.
// the hardware interrupt
Three days into the plan, one of the physical hosts bricked. Cause never fully diagnosed — the decision to swap the chassis came before the diagnosis did, because the migration was the priority and the host was replaceable.
Ten days of pause while a replacement was sourced and racked.
One thing came out of it worth keeping: the prepared control-plane VM survived the host swap intact. Its disks moved to the new chassis and it booted with all the preparation work from a week earlier still in place. That pattern — prep the VM, treat the host as fungible — has been reusable every time hardware has moved since.
// measuring the failover
Three control planes joined, etcd at three members, virtual IP with three-way failover. On paper, high availability.
On paper is not evidence. So the next session was spent deliberately breaking it: hard-stopping each control plane in turn at the hypervisor, with no graceful shutdown, and measuring what actually happened.
| Test | API endpoint outage | Quorum | Recovery |
|---|---|---|---|
| Primary control plane killed | ~0s observed | 2 of 3 held | clean, ~90s |
| Second control plane killed | ~8s | 2 of 3 held | clean, ~90s |
| Third control plane killed | ~4s | 2 of 3 held | clean, ~90s |
Under ten seconds in every case. Tenant workloads were unaffected throughout.
Two things worth noting. The first test's apparent zero-second outage was a monitoring artefact — the monitor was running somewhere affected by the test. Where you put the observer matters as much as what you're observing.
And hard-stopping at the hypervisor produces a sharper failure than stopping the service. A graceful shutdown lets everything deregister politely. That is not the failure you are worried about.
I have not seen these numbers published for a Kubernetes cluster this size. They are better than I expected.
// six minor versions in a week
The plan allowed one morning per Kubernetes minor version with soak days in between. Under high availability it was so uneventful that the upgrade ran past its own target and kept going — six minor jumps inside a calendar week, control planes one at a time, workers in batches, cluster never below two functioning control planes.
The lesson is narrower than "Kubernetes upgrades are easy." It is that the upgrade was never the risk. The risk was doing it without a control plane that could lose a member. Once that was true, the thing everyone dreads became routine.
// sixty-seven days
Then the audit submission needed persistent storage, and the storage problem was still there.
The symptom: the inventory endpoint returned an empty storage array. Every other signal said storage was fine. The Ceph cluster was healthy with 19 TiB raw. Persistent volume claims bound normally when tested. The StorageClass carried both documented annotations. The provider simply declined every order that asked for persistent storage, reporting no capacity.
What had been tried across those sixty-seven days: adjusting the annotations, restarting the inventory operator, downgrading it, removing a broken liveness probe, and a complete provider version migration undertaken partly on the assumption that it would resolve. It did not. A post in the community provider channel got no answer.
What broke it was cloning the provider source at the exact version deployed and reading the inventory operator's storage aggregation.
The code discovers every StorageClass, builds a map, and then filters that map before reporting. The filter tests a boolean read from a bare-key label on the StorageClass. Not one of the two documented annotations — those are downstream metadata and gate nothing. A different key, a different metadata type, same namespace.
The StorageClass had both annotations and not the label. So the operator discovered the class, tracked it, and dropped it during aggregation. No error. No warning. Not even a debug line. Just an empty array.
The fix was one command:
kubectl label storageclass <name> akash.network=true
Five seconds later the inventory endpoint reported full pool capacity. Within one bid cycle the provider was bidding on persistent-storage orders again. The audit was submitted the next day and passed all seven benchmarks.
Sixty-seven days. One label. Roughly two hours of actually reading the source.
The lesson I wrote down at the time was: always read the actual source code when a known issue persists across versions. The honest version is the note I added underneath it — should have done this sixty days earlier.
That same bug has since produced two upstream contributions: a documentation issue arguing the fix should be reachable from the symptom rather than only from the install sequence, and a later issue about the silence itself. The operator still drops unlabelled storage classes without saying so.
// hardening
The three months after the audit recovery were spent on things the recovery had exposed as overdue.
Certificates moved to automated renewal — both the provider certificate and the wildcard ingress certificate, with a file-watcher reload path so renewal does not restart the provider. That one has its own write-up and I won't retell it here, but the failure mode behind it is worth stating plainly because it recurs later in this piece: chart defaults silently reassert themselves on every upgrade unless explicitly overridden. That is now a documented pre-flight check rather than something to remember.
Anti-mining enforcement was formalised into three layers and published. The blacklist that drives it is public, with its evidence standards written before the first entry rather than after the first dispute, and with an honest provenance note about which entries carry per-entry evidence and which do not.
And the operational record started being published monthly. The argument for that is simple: a tenant deciding where to deploy has almost nothing to go on beyond on-chain attributes and a price. A provider that publishes its own record — including the parts that went badly — is easier to evaluate than one that doesn't. The secondary benefit turned out to matter more. A monthly report forces a reckoning with the things that are quietly not working, on a schedule, whether or not you feel like having it.
// a silent two weeks
One of the twenty-four nodes held Ready for two weeks with no working network path for its pods.
Kubernetes said the node was fine. The kubelet was reporting heartbeats on schedule, the pods assigned to it showed as scheduled and healthy, nothing was being restarted. The signal Kubernetes watches is its own control-plane connectivity to the kubelet, which was intact throughout.
What was broken was the data-plane. Pods on that node couldn't reach storage, couldn't reach each other, couldn't reach anything outside — and the signal that would have shown it never travelled through the kubelet, because the kubelet wasn't the thing broken.
It was caught by cross-referencing two independent sources of truth: the Kubernetes node list against the storage cluster's own view of which hosts its daemons were reachable from. A node that holds Ready on one side but is unreachable from the other is in a split state, by definition, and that cross-reference is now one of the handful of checks that run on a schedule.
Nothing had alerted during the two weeks. The failure mode was the shape every expensive incident in these six months carried — a thing that stopped being true, with nothing in the stack built to notice.
// the rebuild
August and September were physical.
The provider estate had been running on a single-core legacy switching design. One switch failure took the entire provider. Any change to that core meant a full outage, which is why several improvements had stayed deferred for months — the cost of touching it was always higher than the cost of not touching it, right up until it wasn't.
The programme ran in three phases over a month: relocating the small-form-factor node estate to new rack space, then the hypervisors one at a time, then the network cutover itself — twenty-four hosts moved from the legacy gateway to a redundant gateway pair on a dual-TOR fabric, workers first, control planes second, the primary control plane last. One near-miss from the broader rack programme — on the validator side, not the provider side — has its own write-up.
Zero leases lost across the entire programme. Twenty-four nodes, twenty storage daemons, three etcd members, three storage monitors, all relocated.
Three things from that month I would not want to rediscover.
noout is the wrong tool for a rack. It is the obvious-looking shortcut for a maintenance window — tell the storage cluster not to rebalance while hosts are down. For a single host, briefly, it is correct. For multiple hosts under a three-replica policy with host-level failure domain, it would have left roughly forty percent of placement groups inactive. The obvious shortcut and the correct action diverge sharply at scale.
A running component can be in an unrecoverable state without being unhealthy. A GPU operator had applied a scheduling taint to one workstation. A storage monitor and a storage daemon were running there perfectly — and could never have been rescheduled if they stopped. Nothing alerted. It surfaced only when I tried to move them deliberately. Healthy and recoverable are different properties and only one of them is monitored.
Diagnose from state, not from logs. Twice in the programme a storage daemon was diagnosed from heartbeat log text, and twice it was the wrong daemon. The actual fault both times was a node losing power to a failing cable. One command listing daemon state gives the answer directly. That is now the first command rather than the third.
There was also a switch whose running configuration had never been written to startup — a reload would have taken the configuration with it — and two marginal ports with around twenty thousand accumulated receive errors, which on recabling retrospectively explained three unattributed incidents from earlier in the year. Intermittent physical faults present as software mysteries for as long as you let them.
// four days of nothing
During the network cutover the provider created no bids for four days.
It recovered without intervention. No leases were lost — existing workloads ran throughout; only new bidding stopped. Nothing alerted, because nothing had failed. There was no error to catch.
It was found afterwards, by counting transactions.
This is the part worth sitting with. A provider that has stopped bidding looks identical, on every dashboard available to its operator, to a provider that is bidding and simply not winning. Both show healthy pods, healthy storage, healthy nodes, active leases ticking over. The difference between them is a thing that is not happening, and monitoring is built to catch things that happen.
The same month's provider upgrade revealed a longer-running version of the same problem. Three previous upgrades had failed outright — and the provider had kept running on its last good configuration the entire time, looking healthy by every external check. Worse, each subsequent upgrade correctly built on the last successfully deployed configuration, silently discarding everything in the failed ones.
Six weeks of changes that appeared to have been applied and had not. A certificate setting that never actually changed. An on-chain attribute update that never deployed. Two values that stayed inverted on-chain until they were corrected directly.
Both root causes turned out to be upstream bugs in the deployment chart — one where disabling a feature drops the container's entire environment block and the resulting object is rejected, another where an attribute comparison sorts keys and values independently and therefore cannot see two values that have been swapped. Both are now filed.
// unaudited
In August the attestation was removed again — this time not through any failure on the provider's side. An auditor adjusted the scope of its portfolio and this provider fell outside it. No compliance finding. Nothing about the hardware, the uptime or the pricing changed.
The practical consequence was larger than expected, and took an outside dataset to make visible.
The deployment console's default flow filters to audited providers. So does every trial account. The effect was immediate invisibility to the main discovery path, with no signal on the provider side at all — because the way that invisibility manifests is a bid that never happens.
An independent benchmarking project noticed first. It had been running automated deployments across the network for a month and observed that this provider had stopped bidding on its orders entirely: hundreds of orders broadcast, hundreds of bids drawn from other providers, none from here — while its own market snapshots showed this provider advertising available capacity in one hundred percent of over a thousand samples across the same window.
From inside, there was nothing to see. The bid engine was healthy and transacting normally. Thousands of bids over the period, a failure rate well under one percent, nearly all of those losing a race to the chain's bid cap.
The answer was on-chain. Pulling a live order from that tenant showed a signed_by requirement naming a specific auditor. Every open order from them carried it. Without the attestation, those orders are unmatchable at protocol level — and the bid engine declines, correctly, every time, in silence, because a declined bid writes nothing anywhere.
There is a reciprocal finding in that, which the other party has since accepted and is fixing: their own tooling had stated publicly, more than once, that it did not filter on audit status. On-chain, it did. The requirement had been inherited from a template rather than chosen, and was invisible from inside their own tooling for exactly the same reason the missing bids were invisible from inside mine — both of us were reading our own output rather than what the chain actually said.
At the time of writing the provider is still unaudited. A re-audit request was submitted in October and is outstanding. The connectivity-upgrade dependency that had previously held the request — a WAN capacity upgrade from Optus 800/80 to Aussie Broadband 2000/500 Mbps — has since completed. What remains is the audit response itself.
// the pattern
Four incidents cost real time or real revenue across these six months.
A storage class silently filtered out of inventory for sixty-seven days. Deployment upgrades that failed while reporting success for six weeks. A node that held Ready for two weeks with no working network path. Bids that simply never happened.
Not one of them produced an error. Not one would have been caught by alerting on failures, because none of them was a failure in any form a monitoring system recognises. They were absences — and an absence does not write a log line, does not increment a counter, and does not fire an alert.
What has actually worked against this is narrow and unglamorous. Count things that should be happening and alert when the count is zero, rather than alerting when something breaks. Cross-reference two independent signals and treat divergence as a fault, which is how a silently partitioned node is now detected — with the honest caveat that the check currently covers nineteen of twenty-four nodes, because five carry no storage daemon to cross-reference against, and that gap is still open.
And publish the operational record. Two of the four were found because an outside party was measuring, and one of those was found because they published their measurements and I could check them against my own. That cuts both ways — the monthly reports exist so somebody can do the same to me.
That is the uncomfortable conclusion of six months. The failures that cost the most were not the ones that broke. They were the ones that stopped — and nothing, anywhere, said so.
// the receipts
Everything above is checkable. The repositories are sanitized but not curated — what's in them is what runs.
-
akash-provider-ops-public — operational scripts, monitoring and anti-mining tooling, and the monthly operator reports under
reports/ - akash-miner-blacklist — the community blacklist, its evidence standards, and its provenance disclosure
- Upstream — the storage-label issues (#645, #663) and the two deployment-chart bugs (#595, #596)
The operator reports are published monthly and carry the same structure: what ran, what shipped, what broke. They include the months where the answer to the third one was long.
Operated from Sydney, au-syd region.
Top comments (0)