DEV Community

iapilgrim
iapilgrim

Posted on

GCP The Hard Way — Part 6: Validating Disaster Recovery Assumptions With a Live Drill

#gcp #disasterrecovery #cloudsql #multiregion

Introduction

Disaster recovery plans are frequently validated on paper and rarely
tested end-to-end, which means the first real test often happens
during an actual incident. This post walks through provisioning
cross-region Cloud SQL replication and dual-region storage, then
deliberately removing the primary region's infrastructure to measure
actual Recovery Time Objective (RTO) and Recovery Point Objective
(RPO) against a documented estimate made in advance.

Solution overview 

Region: asia-southeast1 (Primary)      Region: asia-east1 (DR)
┌─────────────────────┐               ┌─────────────────────┐
│  Compute Engine VM    │              │                       │
│  Cloud SQL (Primary)  │──replication▶│  Cloud SQL (Replica)  │
└─────────────────────┘               └─────────────────────┘
         │                                        
         ▼                                        
┌─────────────────────────────────┐
│  Cloud Storage (dual-region: ASIA1) │
└─────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Prerequisites

  • Two target regions selected for primary and DR roles
  • Cloud DNS zone already configured for the application domain

Walkthrough

Step 1: Document your recovery estimate before building anything

Before provisioning any infrastructure, record a written estimate:

Estimated RTO: ____ minutes
Estimated RPO: ____ seconds/minutes
Anticipated recovery steps: 1. 2. 3.
Enter fullscreen mode Exit fullscreen mode

Do not revise this document until after the drill — the comparison at
the end is the primary value of this exercise.

Step 2: Provision cross-region replication 

gcloud sql instances create app-db-primary \
  --database-version=MYSQL_8_0 --tier=db-g1-small \
  --region=asia-southeast1 --backup --enable-bin-log

gcloud sql instances create app-db-replica-dr \
  --master-instance-name=app-db-primary --region=asia-east1 \
  --tier=db-g1-small
Enter fullscreen mode Exit fullscreen mode

Step 3: Provision dual-region storage 

gcloud storage buckets create gs://<PROJECT_ID>-dr-bucket --location=ASIA1
Enter fullscreen mode Exit fullscreen mode

Step 4: Deploy the application and point DNS at the primary region 

gcloud compute instances create app-vm-primary \
  --zone=asia-southeast1-a --machine-type=e2-small \
  --image-family=debian-12 --image-project=debian-cloud

gcloud dns record-sets create app.yourdomain-test.com. \
  --zone=app-dr-zone --type=A --ttl=300 \
  --rrdatas=<PRIMARY_VM_IP>
Enter fullscreen mode Exit fullscreen mode

A 300-second TTL is used deliberately — higher TTLs common in default
registrar configurations directly extend recovery time by keeping
stale DNS resolutions cached at clients and resolvers.

Step 5: Simulate the disaster

Record the exact start time before proceeding — this is your RTO
measurement baseline:

date -u +"%Y-%m-%dT%H:%M:%SZ" > disaster-start-time.txt

gcloud compute instances delete app-vm-primary --zone=asia-southeast1-a --quiet
gcloud sql instances delete app-db-primary --quiet
Enter fullscreen mode Exit fullscreen mode

Deleting the primary instance also orphans the read replica — a
realistic condition that must be handled as part of recovery.

Step 6: Execute recovery without referencing the original plan 

gcloud sql instances promote-replica app-db-replica-dr
Enter fullscreen mode Exit fullscreen mode

Deploy the application in the DR region:

gcloud compute instances create app-vm-dr \
  --zone=asia-east1-a --machine-type=e2-small \
  --image-family=debian-12 --image-project=debian-cloud
Enter fullscreen mode Exit fullscreen mode

Expected friction point: connection strings hardcoded to the
original instance name will need manual updates once
app-db-replica-dr becomes the new writable primary. This is a direct
argument for externalizing connection configuration (via Secret
Manager or environment variables) rather than embedding it in
application code or images.

Update DNS to point at the DR region, and record the completion time:

gcloud dns record-sets update app.yourdomain-test.com. \
  --zone=app-dr-zone --type=A --ttl=300 --rrdatas=<DR_VM_IP>

date -u +"%Y-%m-%dT%H:%M:%SZ" > recovery-end-time.txt
Enter fullscreen mode Exit fullscreen mode

Verify DNS resolution independently (e.g., with dig) rather than
assuming propagation is instantaneous once the record is updated.

Step 7: Measure RPO

Cloud SQL read replicas replicate asynchronously; any writes to the
primary that had not yet replicated at the moment of deletion are
lost. To quantify this in a follow-up test run, write a record
immediately before deleting the primary and confirm whether it is
present in the promoted replica — this delta is your empirical RPO.

Comparing estimate to reality

Metric Documented estimate Measured result Delta
RTO
RPO

Most teams running this drill for the first time find their initial
estimate omitted at least one dependency — commonly DNS propagation
delay or hardcoded connection configuration.

Clean up resources 

gcloud compute instances delete app-vm-dr --zone=asia-east1-a --quiet
gcloud sql instances delete app-db-replica-dr --quiet
gcloud storage rm -r gs://<PROJECT_ID>-dr-bucket
gcloud dns managed-zones delete app-dr-zone --quiet
Enter fullscreen mode Exit fullscreen mode

Conclusion

Replication and backups reduce data loss risk but do not eliminate
it — RPO is bounded by replication lag, not by the mere existence of a
replica. The most valuable output of this exercise isn't the
architecture itself, but the revised recovery runbook you write
immediately afterward, informed by what your original plan missed.

In Part 7, we shift from availability to cost, examining how
common infrastructure choices translate into avoidable cloud spend.

Top comments (0)