High availability gets treated as a checkbox, "we need HA," without much precision about what that actually means or what it costs to achieve. Let's fix that, because the concept is simple once you strip the buzzword off it, and the design decisions that follow are concrete, not vibes.
HA is a number, not a feeling. Availability is measured as the percentage of time a system is actually up and working, and it's usually expressed in nines. Ninety-nine percent sounds great until you convert it: that's about 3.65 days of downtime a year, which is a lot if the system matters. Ninety-nine point nine percent, three nines, is around 8.7 hours a year. Four nines is about 52 minutes a year. Five nines is around 5 minutes a year. Each additional nine is an order of magnitude harder and more expensive to achieve, and this is the number you should actually be negotiating, not the word "high availability" in the abstract. Before designing anything, get an explicit target: which number of nines does this system actually need, given what downtime costs you.
Redundancy is the mechanism, and it has to be genuinely independent. The core technique for HA is eliminating single points of failure, anything whose failure alone takes the system down. That means duplicate components, and critically, duplicates that fail independently of each other. Two servers in the same rack sharing a power feed aren't independent, they'll die together. Two internet links from the same provider running through the same physical conduit aren't independent, one construction accident kills both. Real redundancy means different failure domains, different power, different paths, different providers where it matters, so a single event can't take out primary and backup at once. This is the detail most HA designs get wrong, they add a backup without verifying it actually fails independently.
Failover has to be automatic and fast, or it doesn't count. Having a redundant component sitting idle is only half of HA. The other half is detection and failover, noticing the primary failed and shifting to the backup, ideally automatically and within seconds, not requiring a human to notice and intervene. Manual failover might be acceptable for a system with a generous downtime budget. It's useless for anything targeting four or five nines, because a human noticing, diagnosing, and manually failing over will blow through your entire annual downtime budget on one incident.
Test your failover, because untested failover is a hypothesis. This is the step almost everyone skips, and it's the one that determines whether your HA design actually works when it matters. Deliberately fail the primary in a controlled way and confirm the backup takes over cleanly, within your target time, without data loss or corruption. Teams that skip this routinely discover, during a real incident, that the failover mechanism had a bug, or a dependency nobody accounted for, or simply never triggered. An HA design that's never been tested under real failure conditions is a design you're hoping works, not one you know works.
Match the target to what downtime actually costs. Here's the counterbalance. Five nines is dramatically more expensive and complex than three nines, and that complexity is itself a risk, an elaborate HA system nobody fully understands can fail in confusing new ways. So don't reach for the highest number by default. Calculate what downtime actually costs this specific system, in money, in user trust, in cascading effects on other systems, and pick a target that matches. A system where an hour of downtime is a minor inconvenience doesn't need the same investment as one where a minute costs real money. Over-engineering HA for a low-stakes system wastes resources you needed for the system that actually deserved five nines.
Design for graceful degradation, not just binary up or down. A more sophisticated version of HA thinking, worth building toward once the basics are solid: instead of a system being either fully up or fully down, design it so partial failures cause partial, contained degradation rather than total collapse. One component struggling should ideally mean some reduced functionality, not everything going dark. This is harder to design than simple redundancy, but it means your failure modes are softer and your actual availability, felt by users, ends up better than the raw number might suggest.
The whole exercise starts with a number, not a feeling, target availability, translated into a downtime budget you can actually reason about. Everything else, redundancy, independent failure domains, automatic and tested failover, sizing the investment to what downtime actually costs, follows from taking that number seriously instead of treating "high availability" as a label you slap on and move past.
Top comments (0)