DEV Community

Cover image for Disaster Recovery on Alibaba Cloud: Designing for Failure
Raphael Gab-Momoh
Raphael Gab-Momoh

Posted on Originally published at raphaelgmomoh.pages.dev

Disaster Recovery on Alibaba Cloud: Designing for Failure

#alibabacloudengineeringlabseri #disasterrecovery #reliability #alibabacloud

Part 8 of the Alibaba Cloud Engineering Lab Series.

Architecture 

Zone A (primary)              Zone B (standby)
  ECS + ACK node pool           ECS + ACK node pool (min-size 0, scales on failover)
  RDS primary  ─────replication─────▶ RDS standby
  OSS bucket (versioning + cross-region replication enabled)
Enter fullscreen mode Exit fullscreen mode

A DR plan that has never been tested is a hypothesis, not a plan. This lab runs three controlled failure experiments and records what actually happened, not what the architecture diagram implies should happen.

Before the how, the what — the two terms every DR conversation revolves around:

  • RTO (Recovery Time Objective) — how long you're willing to be down before service is restored. An RTO of 5 minutes means: from the moment something fails, you have 5 minutes to get back to serving traffic before it's considered a target miss.
  • RPO (Recovery Point Objective) — how much data you're willing to lose, measured in time. An RPO of 15 minutes means your backups/replication are frequent enough that, worst case, you lose the last 15 minutes of writes — not that recovery itself takes 15 minutes.

They're independent numbers answering different questions — "how long until we're back up" versus "how much data are we willing to lose" — and a real DR test measures both separately, which is exactly what this lab does.

Problem

The RTO/RPO targets on the whiteboard were: application failure recoverable in under 1 minute, instance failure in under 5 minutes, data loss recoverable to within 15 minutes of the incident. None of these had ever been tested end-to-end.

Implementation & Experiment

Experiment 1 — Container crash:

kubectl exec -it api-pod-xyz -- kill 1
Enter fullscreen mode Exit fullscreen mode

Experiment 2 — ECS instance failure (simulated by force-stopping the instance backing an ACK node):

aliyun ecs StopInstance --InstanceId i-xxxxxx --ForceStop true
Enter fullscreen mode Exit fullscreen mode

Experiment 3 — Data loss (deleted an OSS object that versioning should protect):

ossutil rm oss://prod-data-bucket/critical-report.json
Enter fullscreen mode Exit fullscreen mode

Failure / Challenge

Experiment 2 was the one that broke the assumption: the ACK cluster autoscaler took 6 minutes 40 seconds to detect the missing node, provision a replacement, and reschedule the evicted pods — well past the 5-minute target. The gap was the node pool's health-check and scale-up interval defaults, tuned for cost-conscious slow scaling, not fast recovery.

Solution

Tightened the node pool's scale-up responsiveness and added a pod disruption budget plus a second, smaller standby node kept warm specifically to absorb single-node failure without waiting for a fresh node to boot:

resource "alicloud_cs_kubernetes_node_pool" "standby_buffer" {
  cluster_id     = alicloud_cs_managed_kubernetes.primary.id
  node_pool_name = "standby-buffer"
  vswitch_ids    = [alicloud_vswitch.zone_a.id]
  instance_types = ["ecs.g6.large"]
  desired_size   = 1 # always-on buffer capacity
  scaling_config {
    min_size = 1
    max_size = 1
  }
}
Enter fullscreen mode Exit fullscreen mode

Re-ran Experiment 2: recovery dropped to 58 seconds — the buffer node absorbed the evicted pods immediately, while a replacement node provisioned in the background to restore the buffer.

Results Table

Scenario Failure Recovery Method Target RTO Actual RTO (before fix) Actual RTO (after fix)
Container crash Process killed Kubernetes restart < 1 min 8 sec 8 sec
ECS/node failure Instance force-stopped Autoscaler + reschedule < 5 min 6 min 40 sec 58 sec
Data loss Object deleted OSS versioning restore < 15 min 3 min 10 sec 3 min 10 sec

The container-crash and data-loss scenarios met target on the first try. The node-failure scenario didn't — and wouldn't have been caught without actually running the experiment instead of trusting the architecture diagram.

Cost / Performance

The always-on standby buffer node adds a fixed cost (~1 extra ecs.g6.large node, ~$95/month) purely for recovery-speed insurance. Whether that tradeoff is worth it depends entirely on what a 6-minute outage actually costs your business — for a payments platform, obviously yes; for an internal reporting dashboard, probably not.

Lessons Learned

  • An RTO target that has never been tested is a guess dressed up as a commitment — run the failure, measure the real number.
  • Autoscaler defaults are tuned for cost efficiency, not recovery speed — the two goals are in tension, and DR requirements should explicitly override cost defaults where they conflict.
  • Not every recovery scenario needs the same investment — spend the always-on buffer cost only on the failure modes whose business impact justifies it.

GitHub Repository: alibaba-cloud-disaster-recovery-lab — the three experiment scripts, Terraform for the DR stack, and the standby-buffer fix, ready to run.

Disaster Recovery · Alibaba Cloud · RTO · RPO · Reliability · Chaos Engineering

Originally published on my portfolio.

Top comments (0)