Every architecture diagram I have ever seen promises resilience. Redundant instances, a standby database, traffic that "automatically fails over" if a zone goes down. It is drawn confidently, in reassuring boxes with arrows, and it is almost always untested. The first time that failover machinery actually runs is during a real outage, under real pressure, and that is precisely when you discover the standby was never quite configured, the DNS took nine minutes to switch, or the replica was silently hours behind.
High availability that has never been exercised is not high availability. It is a theory. And theories about how systems behave under failure are wrong far more often than we like to admit, because the failure modes that matter are exactly the ones nobody thought to design for. The diagram assumes the healthy node notices the sick one and takes over cleanly. Reality serves up the sick node that is not dead enough to trigger failover but too sick to serve traffic, and your beautiful redundancy sits there doing nothing while everything hangs.
The only way I know to trust a failover is to cause one deliberately, on a normal day, when everyone is awake and watching. Kill the primary and confirm the standby actually takes over. Pull a node out of the load balancer and confirm traffic reroutes without dropping requests. Sever the connection to a region and see what really happens, as opposed to what the diagram says should happen. Do it in a controlled way first, then do it in production, because the two environments always differ in the detail that ends up mattering.
This makes people nervous, and I understand why. Intentionally breaking a working system feels reckless. But you are going to run this failover eventually, whether you schedule it or an outage schedules it for you. The only choice is whether the first run happens at 2pm with the whole team ready and a rollback prepared, or at 3am with one exhausted person discovering, live, that the safety net has a hole in it.
Redundancy you have never tested is a cost you are paying for a comfort you do not actually have. Kill a server on purpose. Find out today what you would otherwise find out on your worst night.
– Serguey Shinder
Top comments (0)