DEV Community

Cover image for Multi-AZ Isn't a DR Plan: Zone Outage on OpenShift and AWS
Nerav Doshi
Nerav Doshi

Posted on

Multi-AZ Isn't a DR Plan: Zone Outage on OpenShift and AWS

Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

☁️ Cloud Without the Chaos #6

⚡ Byte Size Summary

  • What a single-zone outage does to a multi-AZ ROSA HCP cluster, and why stateless and stateful workloads fail differently
  • Why zonal storage and load balancer defaults, not the cluster, set your real recovery time
  • A short checklist for testing recovery assumptions before an outage tests them for you

The Story

A customer of mine lived through several AWS zone-level outages in a short stretch. The clusters were multi-AZ, and the working assumption was that this meant they were already resilient. After a few hours of a real zone outage, a platform engineer and the head of IT both started asking whether that assumption held.

It didn't, at least not for everything. The cluster lost 33% of its compute nodes. Stateless services kept running because autoscaling placed replacements in the surviving zones, and they were back in about 25 minutes. Stateful services did not come back on their own. Total downtime for that tier was 3 hours 15 minutes, and most of it was manual work.

The customer runs sensitive applications on-prem and cloud-native applications on a managed platform. That mix made the next conversation harder, because "what is our DR strategy?" had no single answer.

The Problem

Multi-AZ protects you from losing a node or a zone's compute. It does not protect you from the things that are tied to the zone: the volumes your pods are mounted to, the paths your traffic takes in, and the backups you've never restored.

This is blast radius, the fifth dimension of the framework from Cloud Without the Chaos #1. On paper, three zones cap the damage at a third of the cluster. In practice, what was pinned to the failed zone set the blast radius: zonal volumes, cached load balancer addresses, and a backup nobody had restored.

The people who feel this are the platform team on call and the business owner deciding whether to restore from backup. Both are making decisions in the middle of an incident with information nobody verified beforehand.

Why Existing Approaches Fall Short

Most teams treat multi-AZ as the DR plan and stop there. It is high availability, and it covers a different failure class. A regional event can take every zone with it, so a plan that stops at the zone boundary has no answer for that case. Surviving that means either rebuilding the cluster in another region from backups or running active-active across regions, and both choices have to be made before the outage.

A real DR strategy is made at the workload level, not per cluster. Before an incident, a team should be able to answer these:

  • Which workloads need to recover, and in what order?
  • What recovery time and recovery point does each one need?
  • Where does the data live, and how does it get to another zone or region?
  • What are the dependencies outside the cluster, such as DNS, registry and identity?
  • Has any of it been tested end to end, including a restore?

Most teams I see have answers to the first and third questions and guesses at the rest.

The Architecture

Zone outage traffic and storage paths on a multi-AZ ROSA HCP cluster

The diagram follows two paths through the same zone outage: the traffic path and the storage path. In the traffic path, requests keep flowing to the failed zone because of how the load balancer and DNS behave. In the storage path, stateful pods are pinned to a zonal volume that cannot move.

The design decision it makes visible is that the cluster spans three zones and the storage underneath it does not. This was a hosted control plane (HCP) cluster on ROSA. The control plane sits in Red Hat's AWS account, while the worker nodes and volumes sit in the customer's account. The blast radius of the zone outage was therefore the customer's data plane. The control plane stayed healthy throughout, because it runs in Red Hat's account, spread across multiple zones and managed separately from the customer's workers and volumes.

Implementation

This is an incident write-up, so there are no install steps. These are the points where recovery was decided.

Traffic kept going to the dead zone. Cross-zone load balancing was disabled on the network load balancer, upstream DNS had cached the load balancer's IP addresses, and target group health checks took time to mark the dead targets unhealthy. Users saw errors until all three caught up. Rollback consideration: enabling cross-zone balancing changes cost and traffic distribution, so test it in a non-production environment first.

Stateful pods stayed stuck. The volumes were ReadWriteOnce block storage that is zonal. A pod can't attach a volume in a zone that isn't there. Recovery meant force-deleting the stuck pods by hand and restoring the data from older snapshots into new volumes in a surviving zone.

The restore decision was made blind. The head of IT chose to restore from older snapshots. The team knew snapshots ran nightly. It had not verified that the previous night's snapshot completed, and nobody checked its timestamp before recovery started. The recovery point had never been pressure-tested.

The cleanup took two days. Afterwards the team spent two days running custom scripts to reconstruct missing information. How much data was affected was not shared with me, so I won't characterise it.

Security Considerations

During recovery, people get broad access fast. Force-deleting pods and restoring snapshots into new volumes needs elevated rights, and the emergency path is when those rights get granted loosely and never revoked. Use time-bound privilege elevation, grant only the permissions required for recovery, and review and revoke elevated access once the incident is resolved.

Snapshots are also a data store. Restoring from an older snapshot means putting older data, with its older access assumptions, back into production. Know who can create, copy and delete them.

Tradeoffs

What you gain by planning DR at the workload level is a recovery order and a number the business has agreed to. What you give up is time. Someone has to classify every workload, and teams without a DR owner tend to stall at this step.

What you gain from rehearsing a restore is a verified recovery point. What you give up is engineering time and some risk on the day of the test, so the test needs its own rollback plan.

What I'd Do Differently

Looking back, I would want backup completion and backup age verified as part of routine operational checks, rather than having to ask for that information during a 2 a.m. incident. I would also define and document the recovery time objective (RTO) and recovery point objective (RPO) for each workload before an outage occurs. Finally, I would include load balancer and DNS failover in regular zone-failure drills, since these behaviours can be tested ahead of time without waiting for a real outage.


GitHub Repo

There is no companion repository for this article. It documents an incident and a decision, not a shareable implementation.


What's Next

Cloud Without the Chaos #7: Applying the Framework: OpenShift on ARO vs. Self-Managed vs. AWS/GCP

Top comments (0)