Fail over a database-backed application between cloud regions by fencing the old primary so it cannot accept writes, promoting a replica of the database in the healthy region, pointing the application at the new primary, and then moving traffic. The database option you choose decides how much data you can lose and whether the application’s endpoint changes.
Disclosure: Control Plane publishes this guide.
TL;DR
Fence the old primary to prevent split-brain, promote the standby database in the healthy region, repoint the application, and shift traffic.
- Order of operations matters. Fence before you promote. Promote before you repoint. Repoint before you shift traffic. Each step depends on the one before it.
- Pick your database option based on RPO and endpoint behavior. Synchronous or consensus-based systems approach zero data loss but add write latency. Asynchronous systems are simpler but can lose recent transactions on failover. Some options keep a stable endpoint; others hand you a new connection string.
- State outside the database has to fail over too. Object storage, secrets, caches, queues, sessions, scheduled jobs, container images, and DNS and certificates all need a plan.
- Fail back as a planned switchover. Rebuild the old primary as a replica, let it catch up, then switch.
- Control Plane fits by running the application tier active-active across regions and clouds, with multi-location database templates in Control Plane’s Template Catalog, its library of ready-to-install apps.
How do I fail over a database-backed application between cloud regions?
You fail over by executing a sequence of coordinated steps: detect the regional failure, fence the old primary database, verify replication state, promote the replica, repoint the application’s database connection, scale the application tier in the surviving region, redirect user traffic, and validate. The sequence keeps writes going to exactly one database instance while minimizing downtime and data loss.
Two numbers govern every decision. Your Recovery Time Objective (RTO) is the maximum acceptable downtime, and your Recovery Point Objective (RPO) is the maximum acceptable data loss, measured in time. The database replication mode you choose sets the floor for your RPO, and the automation level of your runbook sets the floor for your RTO.
Four standby patterns define the spectrum of cost and recovery speed:
- Backup and restore. The cheapest option. RTO measured in hours. RPO equals the age of the last backup.
- Pilot light. Core infrastructure runs in the standby region but at minimal scale. RTO measured in tens of minutes.
- Warm standby. A scaled-down copy of the full environment runs continuously. RTO measured in minutes.
- Active-active. Both regions serve traffic. If both also accept writes, RTO approaches zero, but conflict resolution and data consistency become the hard problems.
Which database options support cross-region failover?
Databases differ in how they replicate, how much data they lose on failover, and whether the application’s endpoint changes.
| Option | Replication across regions | Data loss on unplanned regional failover | How the new primary is chosen | Does the app’s database endpoint change? | Works across cloud providers? |
|---|---|---|---|---|---|
| Amazon Aurora Global Database | Storage-based asynchronous replication, typically under 1 second lag | Depends on replication lag | Managed failover via FailoverGlobalCluster API; you choose the target region | The global writer endpoint follows the promoted region | No. AWS only. |
| Amazon RDS cross-region read replica | Asynchronous streaming replication | Depends on replication lag | Manual promotion to a standalone instance | Yes. The promoted instance gets a new endpoint. | No. AWS only. |
| Azure SQL Database failover groups | Asynchronous geo-replication | Depends on replication lag | Customer-managed (recommended) or Microsoft-managed failover policy | No. The read-write listener endpoint stays the same. | No. Azure only. |
| Google Cloud SQL cross-region replica | Asynchronous replication; Enterprise Plus edition supports cross-region disaster recovery replica switchover and failover | Depends on replication lag | Manual promotion, or managed switchover/failover on Enterprise Plus | Yes, unless using the Enterprise Plus write endpoint, a DNS name that follows the primary after replica failover or switchover | No. GCP only. |
| Azure Cosmos DB (multiple write regions) | Multi-region writes with conflict resolution | Writes not yet replicated out of the failed region can be unavailable, or lost if the region is permanently lost | All regions accept writes; no promotion needed | No. The endpoint is global. | No. Azure only. |
| Azure Cosmos DB (single write region, service-managed failover) | Asynchronous replication | Depends on replication lag | Azure selects the failover region based on configured priorities | No. The account endpoint stays the same. | No. Azure only. |
| Amazon DynamoDB global tables | Multi-Region eventual consistency (default, last-writer-wins) or multi-Region strong consistency (exactly three Regions) | Eventual mode: depends on replication delay, usually seconds. Strong mode: none | All regions accept writes; no promotion needed | Per-region endpoints. The application uses its local region’s endpoint after failing over. | No. AWS only. |
| Google Cloud Spanner multi-region | Synchronous, consensus-based replication across regions | No data loss | Automatic. Consensus elects a new leader. | No. The instance endpoint stays the same. | No. GCP only. |
| CockroachDB multi-region | Consensus-based (Raft) replication | No data loss if a majority of replicas survive | Automatic leader election; surviving a region loss needs at least three regions | No. The application connects through a load balancer or connection string that includes multiple nodes. | Yes. Nodes can run on any provider. |
| YugabyteDB | Synchronous Raft consensus within a cluster across regions, or xCluster asynchronous replication between clusters | No data loss (synchronous); depends on replication lag (xCluster) | Automatic leader election (synchronous); manual promotion (xCluster) | No (synchronous); yes, application reconfiguration needed (xCluster) | Yes. Nodes can run on any provider. |
| Self-managed PostgreSQL with Patroni | Asynchronous streaming replication by default; synchronous optional | Depends on replication lag (async); no data loss (sync) | Automatic promotion via Patroni with a consensus store (etcd, ZooKeeper, or Consul) requiring quorum | Depends on your proxy layer. With a leader-routing proxy, no. Without one, yes. | Yes. Runs anywhere PostgreSQL runs. |
| pgEdge | Active-active PostgreSQL 17 with the Spock multi-master extension; last-update-wins conflict resolution | Depends on replication lag; writes not yet replicated from the failed node can be lost | All nodes accept writes; no promotion needed | No. Each node is a full read-write peer. | Yes. Runs anywhere PostgreSQL runs. |
| MySQL async replicas / InnoDB ClusterSet | Asynchronous replication; InnoDB ClusterSet provides managed cross-cluster replication | Depends on replication lag | Manual promotion (async); controlled switchover or emergency failover (ClusterSet) | Yes, unless using MySQL Router configured to follow the primary | Yes. Runs anywhere MySQL runs. |
Managed single-provider databases
Amazon Aurora Global Database, Azure SQL Database failover groups, and Google Cloud SQL cross-region replicas all use asynchronous replication and are confined to their respective clouds. Their advantage is operational simplicity: the cloud provider manages replication, monitoring, and in some cases automatic failover. Azure SQL Database failover groups, Aurora Global Database, and Cloud SQL Enterprise Plus (through its write endpoint) offer stable endpoints that follow the primary, which means your application’s connection string does not change on failover. RDS cross-region read replicas and basic Cloud SQL replicas require you to update the connection string after promotion.
Distributed SQL
Google Cloud Spanner, CockroachDB, and YugabyteDB use consensus-based synchronous replication. Writes are acknowledged only after a majority of replicas agree, which eliminates data loss on failover but adds write latency proportional to the distance between regions. Spanner is GCP-only. CockroachDB and YugabyteDB run on any infrastructure, making them candidates for cross-provider deployments. All three require at least three regions (or zones spread across regions) to survive the loss of one.
Self-managed PostgreSQL and MySQL
PostgreSQL with Patroni and pgEdge represent two ends of the self-managed spectrum. Patroni automates leader election and promotion using a consensus store, but replication is asynchronous by default, so failover can lose recent transactions. pgEdge eliminates the single-writer bottleneck with active-active multi-master replication, but you must design your schema to tolerate last-update-wins conflict resolution. Both run on any infrastructure, including across cloud providers. MySQL with InnoDB ClusterSet offers Patroni-style cluster management for MySQL, with MySQL Router handling connection routing.
What is the step-by-step failover runbook?
Every failover follows the same logical sequence regardless of which database you use.
| Step | What to do | What breaks if you skip it |
|---|---|---|
| 1. Confirm regional failure and decide | Verify the outage covers the region and is lasting beyond a transient network blip. Decide whether to fail over automatically or through a human gate. | False positive triggers an unnecessary failover, causing downtime and possible data loss from an avoidable promotion. |
| 2. Fence the old primary | Stop writes on the old primary. Revoke access, shut down its application tier, or isolate it at the network level. If the old primary is unreachable, fence it from outside by revoking application credentials, cutting its network access, and removing it from the proxy and DNS. | Split-brain. Both regions accept writes. Divergent data is extremely difficult to reconcile. |
| 3. Check replication lag and record what may be missing | Query the replica’s replication position (WAL LSN for PostgreSQL, GTID for MySQL, equivalent for managed services). Document the gap. | You promote a stale replica without knowing how much data is missing, and you cannot inform stakeholders of the actual RPO. |
| 4. Promote the replica | Use the database’s promotion mechanism: API call for managed databases, pg_promote() or Patroni for PostgreSQL, STOP REPLICA; RESET REPLICA ALL; for MySQL, then disable read_only and super_read_only. | The standby region has no writable database. The application cannot process transactions. |
| 5. Point the application at the new primary | Use a stable endpoint, proxy, or DNS alias if available. Otherwise, update the connection string. Refresh application connection pools. | The application keeps trying to connect to the dead primary. All writes fail. |
| 6. Start or scale the application tier in the healthy region | Ensure enough compute capacity in the surviving region to handle the full production load. | The database is ready but the application tier cannot serve traffic, so users still experience an outage. |
| 7. Move traffic | Update DNS records whose TTLs were lowered in advance, with health checks, or shift weight on a global load balancer. | Users continue to be routed to the failed region. DNS caching with long TTLs extends the outage. |
| 8. Validate with real reads and writes | Execute synthetic transactions. Confirm monitoring dashboards show healthy latency, error rates, and throughput. | A silent failure in the new region goes undetected. You declare recovery but users see errors. |
| 9. Restore protection | Build a new replica from the new primary in another region. Re-establish continuous replication. | You are running without a standby. A second failure in the surviving region is now unrecoverable. |
| 10. Fail back later as a planned switchover | Treat failback as a new planned failover with the same safeguards: fence, check lag, promote, repoint, validate. | Rushing failback causes a second unplanned outage or split-brain. |
Split-brain is the most dangerous failure mode in any failover. It occurs when both the old primary and the new primary accept writes simultaneously, creating two divergent copies of the data. Fencing the old primary before promoting the replica is the single most important step in preventing it. Consensus-based systems like CockroachDB, Spanner, and Patroni-managed PostgreSQL with a quorum store reduce split-brain risk by requiring a majority vote before any node can become the leader, but they still need at least three locations to maintain quorum when one is lost.
How do you keep the application’s connection working after promotion?
The smoothest failovers are the ones where the application’s database connection string never changes. Several mechanisms make this possible:
Stable endpoints. Aurora Global Database’s global writer endpoint and Azure SQL Database’s failover group listener endpoint both follow the primary automatically. Your application connects to the same hostname before and after failover.
Proxies and poolers that follow the leader. PgBouncer, HAProxy, and Patroni’s REST API can be combined so that a proxy layer in each region always routes to the current primary. The application connects to the local proxy, and the proxy handles leader discovery.
DNS aliases. A CNAME record pointing to the current primary’s hostname can be updated during failover. This works for any database but depends on DNS TTL propagation, which introduces a window where some clients still connect to the old primary.
Connection pool refresh. Most connection pool libraries cache connections. After a failover, stale connections must be evicted. Validate connections on checkout, enable TCP keepalives, cap client-side DNS caching (for example the JVM’s networkaddress.cache.ttl), and implement retry logic in your application that detects connection errors and re-establishes connections.
Idempotent retries. During the seconds between fencing the old primary and the application reconnecting to the new one, in-flight transactions will fail. Design write operations to be idempotent so that retrying a failed write does not create duplicate records.
What state outside the database has to fail over too?
The database is the hardest piece, and eight other kinds of state can leave the application broken after a successful database failover.
| State | What to do before failover | Failure if ignored |
|---|---|---|
| Object storage (S3, GCS, Azure Blob) | Use cross-region replication or a provider with built-in multi-region buckets. Verify the standby region can read and write to the same logical bucket or a replica. | Uploads fail. Assets and exports are unavailable. |
| Secrets and credentials | Replicate secrets to the standby region’s secret manager, or use a global secrets service. Ensure database credentials for the new primary are pre-provisioned. | The application cannot authenticate to the database, APIs, or third-party services. |
| Caches (Redis, Memcached) | Pre-warm or accept cold-cache performance degradation. Do not rely on cache replication across regions for correctness. Flush entries that may reflect transactions lost in an asynchronous failover. | A thundering herd of cache misses overwhelms the new primary database. |
| Queues and streams (Kafka, SQS, Pub/Sub) | Use multi-region topics or accept that in-flight messages in the failed region are lost. Consumers in the standby region must be subscribed and ready. | Messages are lost or duplicated. Asynchronous workflows stall. |
| Sessions | Store sessions in the database or a replicated store. Alternatively, use stateless tokens (JWTs). | Users are logged out and lose in-progress work. |
| Scheduled jobs and cron | Ensure only one region runs scheduled jobs at a time. Use a distributed lock or a leader-election mechanism. | Duplicate job execution causes double billing, double emails, or data corruption. |
| Container images and configuration | Push images to a registry accessible from the standby region. Replicate configuration (environment variables, feature flags) to the standby. | The application tier cannot start because it cannot pull images or read its configuration. |
| DNS and certificates | Pre-provision TLS certificates valid for the standby region’s endpoints. Ensure DNS records can be updated programmatically with short TTLs. | HTTPS connections fail with certificate errors. DNS changes take too long to propagate. |
How do you fail back safely?
Treat failback as a new planned switchover with its own preparation and validation.
First, extract writes committed on the old primary that never replicated; rebuilding it discards them. If split-brain occurred during the original failover, even briefly, you may also have conflicting rows. Active-active systems like pgEdge and DynamoDB global tables resolve conflicts with last-writer-wins, but active-passive systems require manual inspection or application-level conflict resolution.
Second, rebuild the old primary as a replica of the new primary. For PostgreSQL, this means running pg_basebackup, or pg_rewind if the old primary has wal_log_hints or data checksums enabled and retains WAL since the divergence point. For managed databases, the cloud provider typically handles re-seeding once you add the old region back as a secondary.
Third, let the rebuilt replica catch up completely. Monitor replication lag until it reaches zero or near zero before proceeding.
Fourth, execute a planned switchover using the same runbook: fence the current primary, verify replication state, promote the replica in the original region, repoint the application, and shift traffic.
Do not rush failback. Running in the failover region for days or weeks is fine. The goal is to restore geographic redundancy first; returning to the original region can wait.
Can a database fail over between cloud providers?
Yes, but only with engine-level replication. Managed cloud databases like Aurora, Cloud SQL, Azure SQL, Cosmos DB, DynamoDB, and Spanner offer cross-region replication and failover only inside their own provider. Reaching another provider needs engine-level replication you run yourself.
Databases that run on your own infrastructure can replicate across providers if the nodes can reach each other over the network. PostgreSQL with Patroni, pgEdge, CockroachDB, YugabyteDB, and MySQL with InnoDB ClusterSet all fall into this category. The hard parts are the network path between providers and the latency it introduces. Cross-provider replication typically runs over the public internet or through dedicated interconnects, and the added round-trip time affects synchronous replication latency and asynchronous replication lag.
The application tier has its own cross-provider routing problem, covered in how to fail over an application across cloud providers automatically.
How Control Plane fits
Control Plane is the unbreakable platform for running production workloads. It runs serverless, standard, stateful, and cron workloads natively on AWS, GCP, Azure, and Oracle Cloud Infrastructure (OCI), VM workloads on locations joined through Control Plane’s Managed Kubernetes (MK8s), and workloads on Kubernetes clusters you already run or your own machines, including on premises, active-active across regions and clouds.
A workload deployed to Control Plane’s Global Virtual Cloud (GVC), a named set of locations, runs in every location of that GVC from one workload definition. Locations can be Control Plane-managed regions on any of the four clouds, Kubernetes clusters you already run joined through Control Plane’s Bring Your Own Kubernetes (BYOK), or your own servers joined through MK8s. Every workload has its own canonical endpoint that DNS geo-routes clients to the nearest location where that workload passes its readiness checks. During the October 20, 2025 AWS us-east-1 outage, Control Plane customers with us-east-1 in their GVC had workloads active and running in healthy locations within 10 seconds, with no customer downtime, according to Control Plane’s internal incident telemetry.
Workload-to-workload traffic uses mTLS with a unique client certificate per workload, behind a deny-by-default internal firewall. Workloads in different clouds reach each other without VPC peering or VPN tunnels you manage. This is what lets engine-level database replication run between locations on different clouds.
Control Plane’s Template Catalog includes three database templates designed for multi-location deployments:
- Postgres Multi-Location. One PostgreSQL 17 cluster managed by Patroni whose members span several locations, with a single primary that takes all writes and asynchronous streaming replicas in other locations. A bundled etcd cluster with one member per location provides consensus. With three or more locations, losing the primary’s location triggers automatic promotion of a replica in a surviving location, and a leader-routing proxy in every location re-routes to the new primary, so clients need no reconfiguration. Because replication is asynchronous, a failover can lose the most recent transactions that had not reached the promoted replica. With two locations there is no automatic failover, and the survivor stays read-only until promoted by hand. Three locations is the recommended minimum.
- pgEdge. Active-active PostgreSQL 17 with the Spock multi-master replication extension across the locations you list. Every node accepts reads and writes, and conflicts resolve with last-update-wins. A per-location HAProxy failover tier and PgBouncer pooler sit in front.
- CockroachDB. A distributed SQL cluster with nodes in the locations you list. Surviving the loss of a region needs three or more locations.
Backups for these templates go to object storage you own (S3, GCS, or MinIO where the template supports it). Control Plane handles Day 2 operations for you, including autoscaling, built-in observability, and security. The 99.999% availability SLA applies to workloads running at least two replicas in at least two locations. Control Plane is PCI DSS Level 1, SOC 2 Type II, HIPAA, and GDPR compliant.
Control Plane does not make asynchronous replication lossless, does not replicate managed cloud databases (Aurora, Cloud SQL, Azure SQL) between providers, and managed databases you keep on a cloud stay on that cloud’s bill. Platforms that fail over the application tier across clouds are compared in the best multi-cloud failover solutions roundup.
FAQ
Short answers to common questions about cross-region database failover.
How do I fail over a database-backed application between cloud regions?
Fence the old primary to prevent split-brain, promote a replica of the database in the healthy region, point the application at the new primary (using a stable endpoint, proxy, or updated connection string), scale the application tier, and shift user traffic with DNS or a global load balancer. The application side (routing, regions, and stateful workloads) is covered in how to run an app across multiple regions.
What is split-brain and how do I prevent it?
Split-brain occurs when two database instances in different regions both accept writes simultaneously, creating divergent copies of your data. Prevent it by fencing the old primary before promoting the replica. Systems that elect a leader by quorum, such as CockroachDB, add a second safeguard.
How much data can I lose in a cross-region database failover?
It depends on the replication mode. Synchronous and consensus-based systems (Spanner, CockroachDB, YugabyteDB in synchronous mode) lose no committed data as long as a majority of replicas survive. Asynchronous systems (Aurora Global Database, RDS cross-region replicas, Cloud SQL replicas, PostgreSQL streaming replication) can lose transactions that were committed on the primary but not yet replicated to the standby. The amount depends on replication lag at the moment of failure. Monitor replication lag continuously and set alerts so you know your actual RPO at any given time.
Should database failover be automatic or manual?
Automatic failover minimizes RTO but risks false positives and split-brain if the detection logic is wrong. Manual failover is slower but lets a human verify the situation before committing. A semi-automatic approach, where automation detects the failure and prepares the environment but a human approves the promotion, is a common pattern for cross-region failover. Within a single region or availability zone, automatic failover is standard and well-tested. Across regions, the blast radius of a false positive is larger, so most teams add a human gate.
Does Aurora Global Database fail over automatically?
Aurora Global Database supports planned switchover through the SwitchoverGlobalCluster API and unplanned failover through the FailoverGlobalCluster API with the allow-data-loss option. You initiate the failover manually or through automation you build. Aurora does not automatically detect a regional outage and fail over on its own without your intervention.
How do I fail over PostgreSQL between regions?
For self-managed PostgreSQL, set up streaming replication to a standby in another region and use Patroni with a distributed consensus store (etcd, ZooKeeper, or Consul) for automatic leader election. With three or more regions in the consensus store, Patroni can promote a replica automatically when the primary’s location is lost. Place a leader-routing proxy (such as HAProxy configured via Patroni’s REST API) in each region so the application always connects to the current primary. Deployment options are compared in best platforms for running Postgres in containers. Compare the cost of active-active versus active-passive before choosing a multi-master design.
What DNS TTL should I use for failover?
Set DNS TTLs for your application’s public endpoint to 60 seconds or less if you plan to use DNS-based failover. Shorter TTLs mean clients pick up the new IP address faster after a record change. Be aware that some resolvers and clients ignore TTLs, so DNS alone is not sufficient for instant failover. A global load balancer with health checks provides faster traffic shifting than DNS record updates. For internal service-to-service traffic, use a proxy or service mesh that discovers the leader directly.
How do I test database failover without downtime?
Run failover drills in a staging environment that mirrors production topology, including cross-region replication and DNS configuration. In production, use controlled chaos engineering: test failover without taking production down by isolating a replica, promoting it, validating writes, and then tearing it down, all without touching the real primary. For managed databases, use the provider’s test failover features (Aurora’s switchover, Azure SQL’s planned failover). Schedule regular drills and document the results, including actual RTO and RPO achieved.
How does Control Plane handle database failover across regions?
Control Plane runs the application tier in every location of a GVC, with a canonical endpoint per workload that DNS geo-routes to the nearest location passing its readiness checks. For the database tier, Control Plane’s Template Catalog provides Postgres Multi-Location (Patroni-managed PostgreSQL with automatic promotion across three or more locations), pgEdge (active-active multi-master PostgreSQL), and CockroachDB (distributed SQL with consensus). Because workloads in different clouds reach each other over mTLS without VPC peering, engine-level replication can run between locations on different clouds. Asynchronous replication can still lose the most recent transactions on failover.
To run a database-backed application across regions and clouds, talk to the Control Plane team or sign up free.
Originally published on the Control Plane blog.
Top comments (0)