Our recovery design was conventional and, on paper, sound. Data replicated continuously to a second region, infrastructure defined as code so it could be built on demand, and a documented sequence for standing up the estate there. We had rehearsed it twice, both times successfully, both times with a reduced set of workloads because a full rehearsal would have been expensive.
The gap opened when a colleague asked what happens if the region we are leaving fails for everybody at once, which is the only realistic reason we would be leaving it.
We tested for that in the only way available, by attempting to build the full production footprint in the recovery region on an ordinary Wednesday. Three things went wrong before we reached the application layer. The instance family our database runs on was not available in the size we needed in that region and would have required a different family with different performance characteristics, which nobody had validated. Our service quotas there were still at the defaults, because we had only ever built a fraction of the estate, and the increase request that fixes this is handled by a support process measured in hours to days. And two of our software licences were registered against the primary region, requiring a call to a vendor whose support desk keeps business hours in another country.
None of that would have mattered in a rehearsal. All of it matters in an event where every other customer in the region is attempting the same build in the same fifteen minutes, and capacity in a cloud region is finite in exactly the way the marketing does not emphasise.
The changes were mostly commercial rather than technical. We now hold reserved capacity in the recovery region for the minimum viable service, which is roughly a quarter of the estate, and we treat the cost as an insurance premium rather than as waste. Quotas there are raised to production levels and checked quarterly. Licences are portable or duplicated. And we have written down, and had signed, which systems we are choosing not to recover in the first forty-eight hours, because that decision is going to be made either way.
Capacity you have not reserved is not a plan. It is a bet that the emergency will be yours alone.
– Serguey Shinder
Top comments (0)