Automatic cross-cloud failover keeps your application available when an entire cloud provider or region goes down. Run the application in both providers, put a health-checked traffic layer in front that moves requests automatically when one provider fails, and replicate data so the surviving side can serve writes; how fast users recover depends on detection interval, failure threshold, DNS caching, and data promotion. This guide walks through every step, compares the major traffic-switching methods with worked timing calculations drawn from vendor documentation, and covers the data, automation, and testing work that makes failover reliable.
Disclosure: Control Plane publishes this guide.
TL;DR
Run active-active or a hot standby in a second provider, front it with DNS failover, a global load balancer, or a proxied anycast endpoint, and replicate data continuously.
Control Plane is the unbreakable platform for running production workloads. Each workload’s built-in canonical endpoint uses DNS to route clients to the nearest location where the workload passes its readiness probes, across AWS, GCP, Azure, and your own Kubernetes clusters, from a single workload definition.
- Failover models range from active-active (no promotion step) to backup and restore (tens of minutes to hours). Choose based on your RTO (recovery time objective, the downtime you can tolerate) and budget.
- Traffic layer determines how fast new requests move. DNS-based methods are bounded by TTL and resolver caching. Proxied methods avoid DNS caching delay.
- Timeline formula: detection time + decision and automation time + propagation time + data promotion time = time until new requests reach the healthy cloud.
- Data replication must be continuous. Asynchronous replication is simpler but risks some data loss. Distributed SQL databases like CockroachDB and YugabyteDB give zero RPO (recovery point objective, the data you can afford to lose) for committed writes when replicas span at least three locations.
- Automation covers monitoring, detection, fencing, readiness checks, data promotion, traffic switch, verification, and notification. Every step must be codified.
- Failback should be a separate, deliberate operation. Some DNS failover services return traffic automatically unless you stop them.
- Testing through regular game days and chaos experiments is the most reliable way to confirm failover works.
How do you fail over an application across cloud providers automatically?
Automatic failover is an eight-step sequence that runs without human intervention. Each step must complete before the next begins.
- Monitor from multiple locations. Health checkers in geographically distributed points of presence send probes to your application’s endpoints. Using multiple locations prevents a single network path failure from triggering a false failover.
- Detect sustained failure. The system marks an endpoint unhealthy only after a configurable number of consecutive failed checks, typically one to ten, with defaults of two or three. The threshold trades speed against false-positive risk.
- Fence the failed side. The automation ensures the old primary can no longer accept writes before anything is promoted.
- Check that the secondary is ready. Before switching traffic, confirm the target cloud is healthy. DNS and load-balancer services only check the secondary’s health endpoint, so capacity and dependency checks belong in that endpoint or in your own automation.
- Promote data if needed. If the database uses asynchronous replication with a standby replica, the replica must be promoted to primary before the application can serve writes.
- Switch traffic. The traffic layer updates its routing. For DNS-based methods, this means publishing new DNS records. For proxied methods, the change takes effect at the proxy edge without waiting for DNS caches to expire.
- Verify with real requests. Synthetic checks or canary requests confirm that the new primary is serving correct responses.
- Notify operators. Alerts fire so the team knows failover occurred, can investigate root cause, and can plan failback.
Failback, the process of returning traffic to the original provider after it recovers, is always a separate, deliberate step. Automatic failback risks sending traffic back to a provider that has only partially recovered, causing a second outage. Plan and execute failback manually or with an explicit approval gate. Route 53 failover records and Traffic Manager priority routing return traffic to the primary automatically once it passes health checks again; to keep failback deliberate, invert the primary’s Route 53 health check (a disabled health check counts as healthy, so disabling it triggers failback), or disable the Traffic Manager endpoint, until you are ready.
For a deeper look at the full lifecycle of running an application across regions and providers, see running an app across multiple regions with routing, databases, and failover.
Which failover model should you choose?
The right model depends on how much downtime your business can tolerate and how much you are willing to spend on standby infrastructure.
| Model | What runs in the second cloud | Typical recovery time | Relative cost | Fits when |
|---|---|---|---|---|
| Active-active | Full application stack and data layer, serving live traffic | Seconds to a few minutes; no promotion step, so the traffic layer sets the time | Highest (see active-active vs. active-passive cost comparison) | Downtime must stay in seconds and the budget supports it |
| Hot standby | Full application stack running idle or at low traffic, with continuously replicated data | Minutes; promotion time plus the traffic layer | High | RTO under five minutes, but you want simpler data routing than active-active |
| Warm standby | Scaled-down application stack with replicated data | Minutes to tens of minutes | Moderate | RTO under 30 minutes, cost matters more than speed |
| Backup and restore | Infrastructure as code templates and backups; nothing running | Tens of minutes to hours | Lowest | Non-critical workloads or disaster recovery as a last resort |
Active-active is the only model where both sides serve full production traffic simultaneously. Every other model requires some promotion or scaling step that adds time. If your RTO is measured in seconds, active-active is the most dependable choice, because nothing has to be promoted or scaled before the second cloud serves traffic.
How long does automatic cross-cloud failover take?
Failover time depends on your traffic-switching method, its configuration, and whether DNS caching is in the path.
Time until new requests reach the healthy cloud = detection time (check interval × consecutive failures required, plus probe timeout on the final check) + decision and automation time + propagation time (DNS TTL plus resolver and client caching for DNS-based methods) + data promotion time (if the database must be promoted).
With DNS-based methods, clients keep existing and pooled keep-alive TCP connections to the failed provider until they fail or close, then reconnect using a fresh DNS answer. With a proxied method, the client connection ends at the proxy edge, so later requests on it can reach the healthy origin, although requests already in flight to the failed origin can still fail.
Third-party rows use settings and behaviors documented by each vendor. The Control Plane row is an observed result from a real outage and measures how fast workloads were active in healthy locations.
| Method | Settings used | Detection | Propagation | New requests move after (approximate) | Notes |
|---|---|---|---|---|---|
| Route 53 DNS failover (standard) | 30 s interval, failure threshold 3 (the default), TTL 60 s | ~90 s (3 × 30 s, plus up to 6 s for the final check’s connect and response timeouts); Route 53 marks the endpoint unhealthy once 18% or fewer of its health checkers report it healthy | Up to 60 s (TTL) | ~2.5 minutes | HTTP checks must connect within 4 s and return 2xx or 3xx within 2 s. Resolver caching can extend propagation. |
| Route 53 DNS failover (fast) | 10 s interval (extra charge), failure threshold 3, TTL 60 s | ~30 s (3 × 10 s) | Up to 60 s (TTL) | ~1.5 minutes | Faster detection; DNS propagation is unchanged. |
| Azure Traffic Manager | 30 s probing, 3 tolerated failures, 10 s probe timeout, TTL 30 s | ~120 s; the endpoint is marked Degraded after the fourth consecutive failure (Microsoft’s monitoring documentation) | Up to 30 s (TTL) | ~2.5 to 3 minutes | Microsoft’s documented example uses these settings. Adding the 10 s probe timeout on the final check gives about 130 s of detection, or about 160 s end to end with the TTL. |
| Cloudflare Load Balancing, proxied (example settings) | 10 s monitor interval (Enterprise minimum), 2 consecutive failures | ~20 to 30 s (2 × 10 s, plus probe timeout and retries) | No DNS caching delay; Cloudflare documents proxied mode as offering faster failover because it is not affected by DNS caching | ~20 to 30 seconds | DNS-only mode adds TTL-based delay. Minimum interval varies by plan: 60 s Pro, 15 s Business, 10 s Enterprise. |
| Control Plane canonical endpoint | Built-in DNS geo-routing to the nearest location where the workload passes its readiness probes, across the locations of Control Plane’s Global Virtual Cloud (GVC), a named set of locations across one or more clouds | Your workload’s readiness probes in each location; no separate health-check service to configure | Handled by Control Plane’s DNS geo-routing; no failover records or TTLs for you to manage | Request-switch time not published; workloads were active and running in healthy locations within 10 seconds (observed October 20, 2025) | During the October 20, 2025 AWS us-east-1 outage, Control Plane customers with us-east-1 in their GVC had workloads active and running in healthy locations within 10 seconds, with no customer downtime, according to Control Plane’s internal incident telemetry. |
Important caveats. DNS resolvers and clients sometimes hold DNS answers longer than the published TTL, so the DNS-based figures in the table are best cases. The data layer can add significant time. Promoting a standby database replica may take seconds to minutes depending on the engine. Replication lag at the moment of failure determines your RPO, the amount of recent data you could lose; promotion time adds to RTO.
For a broader comparison of failover tools, see the multi-cloud failover solutions roundup for 2026.
How do you keep data consistent when failing over between clouds?
Data is the hardest part of cross-cloud failover. Stateless compute moves between clouds easily, while data has to be replicated and kept consistent. The approach you choose determines both your RPO and the complexity of your failover runbook.
| Approach | How writes work | Failover behavior | Data loss risk (RPO) | Watch out for |
|---|---|---|---|---|
| Asynchronous replication with promotion | Writes go to a single primary; a replica in the second cloud lags behind | Replica is promoted to primary; application reconnects | Seconds to minutes of lag, depending on throughput | Promotion can take time; must fence the old primary first |
| Distributed SQL with consensus (CockroachDB, YugabyteDB) | Writes are committed across multiple nodes using Raft consensus | Surviving nodes keep accepting writes if they hold a majority of replicas, which across clouds means nodes in at least three locations | Zero for committed writes while a majority of replicas survives | Cross-cloud latency increases write latency; requires careful node placement |
| Multi-master Postgres with conflict resolution (pgEdge) | Multiple nodes accept writes and replicate asynchronously; conflicts resolved by last-update-wins on commit timestamp | Surviving node continues serving; conflicts resolved after reconnection | Up to the replication lag for writes accepted on the failed node; they replicate, and may conflict, when it reconnects | Conflict resolution logic must be tested thoroughly |
| Partition by tenant or region | Each tenant’s data has its primary in one cloud | Only tenants homed in the failed cloud are affected; they fail over only if their partition is replicated to another cloud | Set by each partition’s replication method; unreplicated partitions are unavailable until the provider recovers | Limits blast radius but provides no failover by itself |
| Backup and restore | Periodic snapshots shipped to the second cloud | Restore from latest backup; provision new database | Up to the backup interval (hours) | Slowest recovery; highest data loss; suitable only for non-critical data |
How do you prevent split brain?
Split brain occurs when both sides of a failover believe they are the primary and accept writes independently, leading to divergent data that is difficult or impossible to reconcile. Three practices prevent it.
Fence the old primary before promoting. Before the standby becomes the new primary, the automation must ensure the old primary can no longer accept writes. This can mean revoking its network access, shutting down the database process, or using a STONITH (shoot the other node in the head) mechanism. When the old provider is unreachable, rely on a lease or quorum rule that makes the old primary stop accepting writes once it loses contact with the witness.
Require a quorum or a witness in a third location. Distributed SQL databases avoid split brain by requiring a majority of nodes to agree on writes. If you use asynchronous replication, place a lightweight witness node in a third location outside both primary clouds so the system can always determine which side has the quorum.
Make the switch decision in one place. The failover controller, whether it is a DNS service, a global load balancer, or a platform, must be the single authority that decides which side is primary. Run that authority outside both providers, or make it quorum-based, so it survives the outage it handles. Two controllers deciding independently cause split brain.
What do you need in place before failover can be automatic?
Automatic failover requires preparation across compute, data, identity, and operations.
- Identical container images in both clouds. The same image must be deployable to any provider without modification. Replicate images to a registry in each cloud so a provider outage does not block pulls.
- Infrastructure as code. Define all infrastructure in version-controlled code, including networking, load balancer configuration, firewall rules, and DNS records.
- Secrets and identity in both providers. Your application needs credentials and service-account bindings in both clouds before failover happens. Pre-provision them.
- Deep health checks that test dependencies. A health check that returns 200 OK without verifying the database, cache, and downstream services will mask failures that matter. Exclude dependencies shared by both sides, or one shared failure marks every location unhealthy.
- Capacity in the surviving cloud for full load. The locations that survive a provider outage must together carry 100% of peak traffic; with two locations, each must carry it alone. See active-active on AWS and Azure without idle standby for the capacity math.
- Runbooks encoded as automation. Script and test every runbook step so no one has to follow a wiki page during an outage.
- Alerts on failover events. Route failover alerts to the on-call channel and include which side is now primary.
- A documented failback procedure. Failback is the reverse of failover and carries its own risks. Document and test it separately.
For a practical walkthrough of adding a second cloud without doubling your operational burden, see how to add a second cloud provider for failover without doubling ops.
How do you test automatic failover?
Untested failover usually fails in ways nobody predicted. Testing must be regular, realistic, and measured.
Game days. Schedule a quarterly exercise where you deliberately fail over production (or a production-equivalent environment) from one cloud to another. Measure actual RTO and RPO. Compare them to your targets. Fix the gaps.
Fail a location. Make one cloud location fail its health checks without changing traffic configuration. Confirm that health checks detect it, traffic shifts, and the application keeps serving correct responses.
Chaos experiments. Inject failures at the network, compute, and data layers. Kill processes, block ports, introduce latency, and revoke credentials. Each experiment reveals a dependency or assumption you did not know about.
Measure time to move traffic. Record the timestamp when the failure begins and the timestamp when the error rate returns to baseline. The difference is your real RTO. If it does not match your target, adjust your health-check intervals, failure thresholds, or failover model.
For a detailed guide on running these tests safely, see how to test failover without taking production down.
How Control Plane automates failover across providers
Control Plane runs serverless, standard, stateful, and cron workloads natively on AWS, GCP, Azure, and Oracle Cloud Infrastructure (OCI). It runs VM workloads on locations joined through Control Plane’s Managed Kubernetes (MK8s), and workloads on your existing Kubernetes clusters or your own machines, including on premises. All of them run active-active across regions and clouds.
A workload deployed to a GVC, which can span one or more cloud providers, runs active-active in every location of that GVC from a single definition. Locations can be Control Plane-managed regions on AWS, GCP, Azure, and Oracle Cloud, your existing Kubernetes clusters joined through Control Plane’s Bring Your Own Kubernetes (BYOK), or servers you supply, on premises or with supported cloud and bare-metal providers, through MK8s.
Each workload’s canonical endpoint, built into Control Plane, uses DNS to route clients to the nearest location where the workload passes its readiness probes, and stops sending new clients to a location once its replicas fail those probes. There is no separate DNS failover service or global load balancer to configure. Control Plane’s location routing options let you set a priority tier per location, a latency offset to fine-tune where traffic lands, and a latency tolerance past which a location is treated as unavailable.
For more on how companies handled major outages, see how companies stay online during AWS us-east-1 outages and the cloud outage tracker.
For data, the Template Catalog, Control Plane’s library of ready-to-install apps, includes multi-location databases such as CockroachDB and pgEdge, so the data layer can survive a provider failure. CockroachDB needs at least three locations to survive the loss of one.
Control Plane’s Universal Cloud Identity gives each workload access to services across AWS, GCP, and Azure through an IAM role or service-account grant instead of long-lived keys in your code, with short-lived, least-privilege credentials at runtime. The same workload can reach provider-specific services in each cloud after failover without credential changes.
The 99.999% availability SLA applies to workloads running at least two replicas in at least two locations. Those locations can be on different providers.
Control Plane handles Day 2 operations for you, including autoscaling, built-in observability, and security. Control Plane is PCI DSS Level 1, SOC 2 Type II, HIPAA, and GDPR compliant. Customers typically cut compute costs 30 to 50 percent compared with running directly on AWS, GCP, or Azure. You pay for the CPU and memory a workload reserves, with no contracts or minimums.
FAQ
These questions cover the most common concerns about automatic cross-cloud failover.
How do you fail over an application across cloud providers automatically?
Run the application in at least two cloud providers with a health-checked traffic layer in front. The traffic layer monitors each provider, detects sustained failure through consecutive failed health checks, and redirects new requests to the healthy provider. Replicate data continuously so the surviving side can serve writes. The eight-step automated sequence is: monitor, detect, fence, check secondary readiness, promote data, switch traffic, verify, and notify. Failback is a separate, deliberate step.
How long does DNS failover take?
DNS failover speed is bounded by the health-check detection time plus DNS TTL and resolver caching. With Amazon Route 53 at a 30-second check interval, threshold of 3, and a 60-second TTL, new requests move in roughly 2.5 minutes. With a 10-second fast interval, that drops to about 1.5 minutes. Azure Traffic Manager with the settings in Microsoft’s documented example (30-second probing, 3 tolerated failures, 10-second timeout, 30-second TTL) takes about 2.5 to 3 minutes. Resolvers that cache beyond the TTL can extend these times.
What is the difference between active-active and active-passive failover?
Active-active runs the full application stack in both providers simultaneously, serving live traffic from both. Failover is fastest because nothing has to be promoted or scaled; the remaining delay is the traffic layer’s detection and any DNS caching. Active-passive (hot, warm, or cold standby) keeps the second provider as a backup that must be promoted or scaled before it can serve traffic. Active-passive is less expensive but slower to recover. See the cost comparison between active-active and active-passive for detailed trade-offs.
Can Route 53 or Azure Traffic Manager fail over to another cloud provider?
Yes. Both services can route to external endpoints by IP address or hostname. Amazon Route 53 health checks can monitor any publicly routable endpoint regardless of where it is hosted; non-AWS endpoints carry an additional health-check charge. Azure Traffic Manager supports external endpoints that point to any FQDN or IP outside Azure. This means either service can direct traffic to an application running on a different cloud provider during failover.
How do I fail over a database between cloud providers?
Choose a replication strategy that matches your RPO. Asynchronous replication with promotion is the simplest: a standby replica in the second cloud streams changes from the primary and is promoted when the primary fails. Distributed SQL databases like CockroachDB and YugabyteDB use Raft consensus to commit writes across clouds, giving zero RPO for committed writes when replicas span at least three locations. Multi-master Postgres solutions like pgEdge allow writes on both sides with conflict resolution. In all cases, fence the old primary before promoting to prevent split brain.
What is split brain and how do I prevent it?
Split brain is a condition where both sides of a failover accept writes independently, creating divergent datasets that are difficult to reconcile. Prevent it by fencing the old primary before promoting the new one, requiring a quorum or witness node in a third location for consensus, and ensuring the failover decision is made by a single authority rather than independent controllers on each side.
How do I fail back after a cloud outage?
Fail back deliberately, never automatically. After the original provider recovers, verify its health independently. First export any writes the old primary accepted that never replicated, then rebuild it as a replica of the current primary, for example with pg_rewind for Postgres, which discards those writes, and reapply or reconcile them. If you use Route 53 failover records or Traffic Manager priority routing, invert the primary’s Route 53 health check or disable the Traffic Manager endpoint until you are ready, because both return traffic automatically. A disabled Route 53 health check counts as healthy. Once data is consistent, use the same traffic-switching mechanism to shift requests back. Test the recovered provider with synthetic traffic before sending real users. Document the failback procedure and test it during game days.
How does Control Plane fail over between cloud providers?
Control Plane deploys workloads active-active across every location in a Global Virtual Cloud from a single definition. Each workload’s canonical endpoint uses DNS to route clients to the nearest location where the workload passes its readiness probes, so clients move to surviving locations, including ones on other providers, when a location fails. Multi-location databases from the Template Catalog cover the data layer, and the 99.999% availability SLA applies to workloads running at least two replicas in at least two locations, which can be on different providers. For examples of how companies handled major outages, see how companies stay online during AWS us-east-1 outages.
Can each location be sized for half of peak traffic?
No. The locations that survive a provider outage must together handle 100% of peak traffic. With two locations, each must carry full peak alone. Provision that capacity in advance, or use autoscaling that reaches it before the shifted load saturates the surviving location, and expect surviving regions to be capacity-constrained during a major outage.
How often should I test failover?
At minimum, quarterly. Game days that simulate a full provider failure are the most realistic test. Supplement them with monthly chaos experiments targeting individual components such as database connections, network paths, and credential access. Measure actual RTO and RPO during every test and compare against your targets.
To run your application active-active across clouds with automatic failover from a single definition, talk to the Control Plane team or sign up free.
Originally published on the Control Plane blog.
Top comments (0)