The plan didn't start at Kava. It started at the Cosmos Hub, because the flagship chain is where you go if you're not sure where to go.
The Cosmos Hub's active set — the top validators eligible for block rewards — has a floor. To enter it, a validator has to hold more staked ATOM than the smallest active validator does. In mid-2026 that floor sits at tens of thousands of ATOM — hundreds of thousands of dollars at current prices. That's an infrastructure business, not a homelab experiment. It's the size of stake where you're raising capital before you're writing a .toml file.
I don't have that capital, and I wasn't going to raise it to prove a point about self-stake. So the plan reshaped itself. Instead of validating on the flagship, target a smaller Cosmos appchain — one where a modest self-stake actually gets into the active set, but the chain is real enough that the validator work isn't performative. Kava won selection as chain #1 on that criterion. Real DeFi product with actual users, Binance-listed, healthy market cap, an active set that was reachable for a modest self-stake at the time I entered. Enough consequence to make the work matter; enough reachability to make the work possible.
Every validator writeup I've ever read assumes you got past this decision on money and skips to the software. This one starts here because for most people evaluating whether to run a validator, this is where the plan actually ends. I want the piece where it doesn't.
One thing the software will not forgive: the same cryptographic key that earns the validator's rewards will, if it ever double-signs a block, get the operator permanently and irrecoverably slashed. Not a warning. Not a temporary jail. The stake and any bonded delegator stake gets burned by the chain itself. Every design decision in what follows is downstream of that constraint.
// the hardware happened to have the right shape
The T430 wasn't acquired for this. It was already in the rack, sitting mostly idle, repurposed from a Filecoin project that didn't pan out. 512 GB of ECC on the box total, most of it committed to what had been the Filecoin VM, and the validator was going to draw from the balance. Because the RAM was there, the sizing wasn't a scarcity negotiation. There was room to be comfortable rather than minimal.
The validator VM got 8 vCPU and 48 GiB. Kava's documented minimums are lower than that. The provision deliberately sat above the documented floor rather than at it, because the downside of under-provisioning a signing node is asymmetric. A resource-starved node misses blocks. Missed blocks eventually jail the validator. Jailed validators lose bonded stake. There is no upside to a lean signing node worth the risk of a starved one, so it got comfortable margin.
The 48 GiB is a fixed allocation. Balloon disabled. Deliberate: the validator's RAM is reserved, not dynamically reclaimable, so memory contention with any other VM on the same host can't starve it. That decision followed the same pattern the Filecoin VM used, which had proven itself.
Storage: a 3.5 TB SSD, passed through to the VM as a dedicated device, mounted at /data/cosmos. Not carved from the shared LVM pool that holds VM OS disks. That separation was an explicit design choice — OS-disk failure and chain-data failure are two independent things that shouldn't share a failure mode. If the OS disk dies, the chain data survives. If the chain data corrupts, the OS is untouched. Two failures stay two failures.
The IOPS discipline was verify-before-trust. Before the node was trusted with signing, the disk was benchmarked: fsync p99 around 734 microseconds, roughly 5,353 IOPS on the workload that mattered. Adequate for the validator's write pattern, confirmed by measurement rather than assumed by spec-sheet reading. The chain is currently running lean on that volume — around 48 GB used of 3.5 TB available — so the storage layer is under no near-term pressure.
That's the hardware. Nothing about it was a targeted purchase for validating. The box had the right shape because the rack had the right box, and the decisions that made the box right for the previous project turned out to make it right for this one too.
// the snapshot that wouldn't replay
The first real war story landed at the snapshot restore step, before signing had ever happened.
Full sync from genesis wasn't the option — the chain history is large enough that starting from block zero would have run the node for days before it caught up. Snapshot restore is the standard alternative: download a recent state snapshot from a public provider, extract it to the chain data directory, start the node from there. Fifteen minutes of work, then the node picks up from a recent height and catches to tip.
The first snapshot source I tried, Polkachu, produced state that wouldn't replay cleanly. The node started, entered replay, and choked partway through with IAVL state that wouldn't reconcile under standard pruning parameters. Three attempts. Each one an evening. Each one ending the same way.
The fix was switching to a different snapshot source — NodeStake — and the replay went clean the first time. Nothing wrong with Polkachu as a provider; the specific snapshot that day just had state my node couldn't reconcile.
What came out of the pain became doctrine: verify the canonical AppHash from a public RPC before committing to a long snapshot mount. Fifteen seconds of check. If the snapshot's AppHash doesn't match what the chain reports at that height, the snapshot is bad — save yourself the hours. That check now runs before any snapshot restore on any chain in the pipeline. It's a five-line addition to a preflight script that would have saved me three evenings on this build.
// Horcrux 2-of-3: the signing architecture
Threshold signing is where the double-sign constraint gets solved.
Horcrux takes the validator's private key and splits it mathematically into shares. Each share, on its own, is useless — it can't sign anything. To produce a valid signature, a configurable threshold of shares must cooperate. The private key never exists as a whole thing on any machine; it only exists as an assembled signature at the moment it signs. There is no file on any host that, if stolen, gives an attacker a working key.
The topology choice is 2-of-3: three cosigners, any two of which are enough to sign. That specific shape is the homelab-scale sweet spot. It buys two properties simultaneously.
Fault tolerance. Any one cosigner can go down — for maintenance, a crash, a host failure — and signing continues on the remaining two. No single machine is a single point of failure for the whole validator.
No single point of compromise. No one machine ever holds a usable key. An attacker who takes one cosigner gets one share, which is worthless without a second. The double-sign threat is neutralised at the architectural level.
Alternative topologies each fail one of those tests. 2-of-2 has no fault tolerance — lose one, signing stops. 3-of-5 buys more redundancy than a homelab cluster with a handful of hosts needs while widening the surface area an attacker has to work through. For a small cluster with a no-slashing-tolerance constraint, 2-of-3 is the minimum topology that gives both properties. That's the honest reason it's the standard homelab-scale answer.
Physical distribution matters as much as the math. The three cosigners run on pro1, pro2, and pro4. The validator's full node runs on pro5. The standby node runs on pro3. No host runs both a cosigner share and a signing node. If any one physical host dies, you lose at most one share — not one share and the node together. Correlated failures at the host level get decorrelated by placement.
The honest nuance: whether every original placement decision was made purely for that reason or partly reflected which hosts had capacity, I can't fully reconstruct. What I can say is that the principle is deliberate — cosigner hosts should be separate from node hosts — and the same principle got applied more explicitly when the second chain (Akash) came online later. It reads as doctrine, and it functions as doctrine, even if the Kava-era placement was principle plus fit rather than principle alone.
The Kava-specific setup went cleanly. The shares distributed on the first attempt, the cosigners came up in the intended order, and the Horcrux Raft cluster formed quorum without incident. The hard-won signing lessons — the startup-ordering rule, the empty-placeholder-key-file gotcha, the strict host-separation rule — surfaced later on the Akash build. Those live in a future piece, not this one.
// the last check before bonding
Before the validator was ever bonded to the network, the check that mattered most was the one that proved Horcrux was actually doing the signing.
The mechanics: Kava's validator identity is normally represented by a file called priv_validator_key.json on the full node. It contains the raw private key the node uses to sign blocks. When Horcrux is doing the signing instead, that file must not be present on the node — otherwise the node would sign locally with its own key, which defeats the entire threshold-signing architecture and reintroduces the single-key-compromise risk.
The discipline is this: archive priv_validator_key.json before going live. Confirm the node is signing through Horcrux with no local key present. Only then bond stake to the validator. Only after signing-under-bond is empirically confirmed — real signed blocks recorded on the chain, produced by the threshold-signing cluster with no local key on the node — is the archived key destroyed.
The reason for that specific ordering is asymmetric downside again. If Horcrux fails silently and the node is still capable of falling back to a local key, the double-sign protection is theoretical rather than real, and the operator won't find out until it matters. If the local key is confirmed absent and the node signs anyway, the protection is real. If the local key is confirmed absent and the node stops signing, that's a fast, safe, revertible failure — you archive-restore the key, diagnose Horcrux, try again. Nobody's stake is at risk during the check.
That check is what "the double-sign will get you permanently slashed" reduces to, operationally. Everything before it is preparation. Everything after it is delegator-visible.
// first signed block, first delegator
The moment the validator was bonded and started signing, I didn't notice.
That's how it's supposed to go. When you've verified everything before going live — the storage, the sizing, the snapshot, the threshold-signing, the local-key discipline — the moment of going live is a non-event. The chain assigns a block. The validator signs it. Then another. Then another. There is no cinematic moment. There is a voting_power field on the node's /status endpoint that reads as a nonzero number instead of zero, and there are heights ticking upward in a monitoring panel, and the operator does not notice because nothing has broken.
That's success. Cinematic first-signed-block moments belong to validators that went live without checking. The absence of drama is what the pre-flight work was for.
The emotional beat, when it did arrive, was three days later. A stranger — someone I had no connection to, no relationship with, no reason to expect — looked at the validator, made an evaluation, and delegated stake to it. First independent delegator. Two total on the validator now, mine plus this one. That's the moment the whole thing stopped being an experiment and became a service someone was using.
Here's the current state, as of writing:
D+28 operational. Uptime 100%. Window block miss: 0 out of 10,000. Two delegators. That's not brag material — it's the operational state, the same way zpool status in the ZFS recovery piece was the state. Numbers on a public chain, verifiable by anyone. The point isn't the numbers. It's that this is what the pre-flight work produces.
// building the safety net: standby + drilled failover
The single node was the launch state. The current state is different.
There's now a standby node on pro3 — a second full node, kept synced to chain tip, ready to take over signing if the primary goes down. Same chain data, same signing configuration, same Horcrux endpoints. The switch from primary to standby is scripted — one script triggered by the operator with a confirmation prompt, another script to fail back to primary once it's healthy again.
The manual failover benchmark, from an earlier phase before automation, was 12 minutes 55 seconds from primary-down to signing-resumed. Human-driven. Every step verified by hand. Correct, but slow.
The scripted failover, drilled against a real deliberately-triggered outage — not a simulation, an actual systemctl stop kava on the primary — clocked at 1 minute 49.5 seconds from primary-down to signing-resumed. 37 blocks missed against the chain's 10,000-block jailing window. Roughly 0.4% of the jailing buffer consumed for a real failover event. Big enough to matter; small enough to survive.
The interesting part of the drill wasn't the failover script — it was the failback. The failback script's first dry run — the one designed to bring primary back into service after standby had taken over — surfaced two real bugs that had never been noticed. A missing privval-blank step, which would have hung real incidents to the 30-minute ceiling every time. A grep -c || echo 0 double-print anti-pattern that produced false WARNING messages downstream. Both were root-caused, fixed, and re-drilled to a clean end-to-end pass before the script was trusted with real operations. That work is exactly what a drill is for — finding bugs in the safety net before the safety net is what stands between you and being jailed.
One additional operational lesson surfaced during the drill: Horcrux's own systemctl ActiveState can report active before its Raft cluster has actually reformed quorum well enough to co-sign. Same category of "healthy-looking status ≠ real capability" issue as the chain's own voting_power field can produce, now confirmed to extend to the signing layer itself. The failback script now waits for genuine signed events, not for ActiveState=active, before declaring signing resumed.
The full topology and drill results live on the validator page. That page has the numbers; this piece has the shape.
// the first governance vote
Three weeks in, the first governance vote came up. Kava proposal 220 — recover the IBC light client for the Kava-Secret channel. Vote was YES.
The operational shape of casting a vote as a validator is different from casting one as a delegator. As a delegator, you click through a wallet UI. As a validator, you sign a governance transaction from the operator account — not the signing key, the operator key — using a discipline the validator page documents as "brief-import." The operator key is not resident on any always-on machine. It gets imported briefly for the vote, the vote transaction is signed and broadcast, and the key is removed again. Same principle as the signing-key discipline, applied to governance: the sensitive credential is never around when it doesn't need to be.
The substantive reasoning on the vote — why yes, what the recurrence of this fix pattern surfaces about relayer-set health on the Kava-Secret channel, why the meta-governance abstain vote is defensible even if it's not what I voted — lives on the on-site governance record. This piece is about the operational experience of casting the vote, not the merits of the proposal. That's a genuine split of concerns: the validator page and the governance record are the accountability artefacts, this piece is the story.
Cross-link to the governance record for the substance: /validators/kava/governance/220/.
// what would have been day-one doctrine
Some of the war stories in this piece became doctrine that got applied to the second chain. If I were starting Kava again from scratch, this is what would be day-one design rather than lesson-learned:
Startup ordering. Horcrux cosigners must be up and connected before the validator node starts its signing daemon. There's a roughly 3-second privval handshake timeout that will crash-loop the daemon if it opens its listener before a cosigner is ready to respond. Every operational script now has cosigner-side up before node-side up as a hard sequence.
The empty-placeholder-key gotcha. Some chain forks panic on startup if the priv_validator_key.json file is present but empty, rather than if it's absent entirely. The safe posture is to remove the file, not zero it. Deleted, not blanked.
AppHash verification before snapshot mount. The Polkachu-snapshot lesson from earlier. Verify the canonical AppHash from a public RPC before starting the long extraction. Fifteen seconds of check, potentially days of saved rework.
Cosigner and node physical separation as strict rule. Not "we should keep them separate where possible" — a hard rule that a cosigner share must never live on the same host as a signing node. Applied to Akash without exception. Would have been applied to Kava the same way if I'd had the discipline at day one.
Drilled failover, not planned failover. A failover script that has never been run against a real outage is theatre. HA4.5's discovery of two real bugs in a script that looked correct is the general case, not the exception. Drill the failover — with a real systemctl stop, not a simulation — before trusting it with a real incident.
None of these are theoretical. Every one of them was learned by doing the work slightly wrong the first time, and none of them cost real slashing because the safety architecture caught the mistakes before they compounded. That's what the safety architecture is for. But knowing what would have been day-one doctrine is the compounding gain — the second chain benefits from every one of these lessons without having to re-learn them.
// closing
Kava is chain #1. Twenty-eight days operational, 100% uptime, two delegators, one governance vote cast. The build wasn't dramatic. The delegator was.
The next chain is Akash. This time the shares distribution is day-one doctrine, not lessons learned later. The startup ordering is day-one doctrine, not a crash loop caught in staging. The snapshot AppHash check runs before the extraction, not after the third failed replay. The failover script gets drilled against a real outage before the validator ever bonds, not weeks after.
The homelab-first framing lands here: the validator playbook doesn't come from cloud infrastructure or a dedicated hosting provider. It comes from the rack, running alongside the rest of the workloads, doing the drilled operational work that most delegators trust cloud providers to handle. Whether that's the right posture for delegators is a conversation for someone else's article. Whether it's a defensible posture for an operator — one where the same person who wrote the failover script has run it, on real hardware, against a real outage — is what the whole series is trying to demonstrate.
Chain #1 signed its first block without ceremony. Chain #2 will sign its first block knowing what the first chain took a month to teach.
Top comments (0)