Denys Zaruin

Posted on Nov 5 Originally published at moveinfra.eu

Why every cloud migration needs a rollback plan, and how to build one

#cloud #cloudmigration #rollbackplan #databasereplication

Originally published on moveinfra.eu.

A cloud migration rollback plan is a documented, pre-tested procedure that lets your engineering team revert traffic and data back to your original cloud provider if serious problems appear during or after cutover. Every migration needs one because production traffic often exposes hidden issues—like network latency spikes, subtle database incompatibilities, or misconfigured firewall rules—that staging tests miss. To build a reliable rollback plan, you must keep your original environment running, lower your DNS time-to-live (TTL) before switching, pause database writes briefly during the cutover, and set up reverse data replication from the new cloud back to the old one so you can switch back days later without losing new customer data.

Key takeaways

Never delete the old environment immediately: Keep your source infrastructure on AWS, Google Cloud, Azure, or DigitalOcean running in a warm standby state for at least 14 to 30 days after cutover.
State is the hardest part of a rollback: Reverting stateless application servers only takes a DNS change, but reverting databases after users have written new data requires reverse logical replication.
Define objective rollback triggers in advance: Agree on clear metrics—such as error rates, database CPU saturation, or payment failures—before the maintenance window begins so tired engineers do not have to debate during an outage.
Test the return path before going live: A rollback plan that exists only in a document is a guess; run a dress rehearsal in staging to verify that reverse replication and DNS changes actually work.
Regulations increasingly require exit and reversal capability: For financial entities and their critical ICT providers in the EU, the Digital Operational Resilience Act (DORA) requires tested exit strategies and continuity plans when switching providers.

Why cloud migrations fail after a successful cutover

Most engineering teams spend 95% of their migration energy getting the new environment running and 5% thinking about what happens if they need to go back. When the new environment passes functional tests in staging, it is tempting to assume the cutover will be final.

In practice, production environments behave differently under real user load. You might switch from a US hyperscaler to an European provider like UpCloud, Scaleway, Hetzner, or OVHcloud and find that everything works for the first three hours. Then a scheduled batch job runs, or Monday morning traffic hits, and an unexpected bottleneck appears.

Common post-cutover failures include:

Connection pool exhaustion: Different managed database configurations or slightly different network round-trip times between application nodes and database nodes cause connection pools to max out under peak load.
Third-party IP allowlists: An external partner, banking API, or enterprise customer firewall silently blocks requests coming from your new cloud's egress IP ranges, even though your team updated the documented integrations.
Storage IOPS limits: Disk read/write performance on the new virtual machines or block storage volumes hits a ceiling during heavy background processing, slowing down the entire application.
TLS and DNS caching quirks: Legacy client applications or corporate proxies ignore standard DNS TTLs and continue sending half of your customer traffic to the old environment while the other half goes to the new one.

If you discover one of these problems ten minutes after switching DNS, fixing it forward might take hours of stressful debugging while your application is down. If you discover it three days after switching, you face a harder problem: your customers have spent three days creating orders, updating profiles, and uploading files in the new cloud. Simply pointing DNS back to the old cloud would erase those three days of business activity.

A complete rollback plan solves both scenarios. It gives you an immediate escape hatch during the cutover window and a safe, data-preserving return path during the weeks that follow.

The two phases of a migration rollback

To design a rollback plan that works in the real world, you need to separate the migration timeline into two distinct risk phases. Each phase has different technical requirements.

Phase 1: Aborting during the cutover window (Point-in-time rollback)

The first phase covers the maintenance or cutover window itself, typically lasting from a few minutes to an hour. During this window, you stop writes to the old database, wait for replication to catch up, run sanity checks on the new environment, and switch your DNS or load balancer traffic.

If something breaks before you allow real users to write data to the new database, rolling back is straightforward. Because the old database still holds the latest authoritative state, you only need to re-enable writes on the old database and keep your DNS pointing at the old cloud. No data is lost, and the old system resumes where it left off.

Phase 2: Reverting after live traffic hits (Stateful rollback)

The second phase starts the exact second a user or background job writes new data to the target cloud. From this moment on, your new EU cloud is the system of record, and your old cloud is out of date.

If a critical flaw appears 48 hours later—for example, a weekly reporting query crashes the database or an undocumented compliance requirement surfaces—you cannot just flip DNS back. To roll back safely during Phase 2, your old database and object storage must continuously receive every new write that happens in the new cloud.

How to build a stateful rollback plan step by step

Building a reliable rollback architecture requires preparing your networks, applications, databases, and DNS well before cutover day. Here is how to set up each layer.

1. Lower DNS TTLs at least 48 hours early

DNS records have a Time-To-Live (TTL) setting that tells recursive resolvers and user devices how long to cache an IP address before asking for a fresh one. If your production domain has a TTL of 3600 seconds (one hour) or 86400 seconds (24 hours), changing your DNS record during an emergency will not move traffic immediately. Many users will stay stuck on the broken environment until their local cache expires.

Two to three days before your migration, lower the TTL on all relevant A, AAAA, and CNAME records to 60 seconds (or 300 seconds if your DNS provider enforces a higher minimum). If you use a proxy service like Cloudflare or an external global load balancer, traffic shifts happen inside the proxy layer within seconds, which gives you much tighter control over both cutover and rollback.

2. Keep your CI/CD pipeline ready for dual deployment

During the transition and rollback window, your application code should be identical in both clouds. Do not combine a cloud infrastructure migration with a major application rewrite or database version upgrade.

Configure your CI/CD pipeline (such as GitHub Actions or GitLab CI) so that you can deploy the exact same container image or build artifact to both the old cloud and the new cloud. If you have to push an emergency bug fix two days after cutover, deploy it to the old environment too so your standby infrastructure stays ready to receive traffic.

3. Configure reverse database replication

This is the technical core of a Phase 2 rollback. During the initial migration, you likely set up logical replication from your old cloud (source) to your new cloud (target) so data stays in sync up to the cutover moment.

To enable a safe rollback after cutover, you must prepare the reverse path: logical replication from the new database back to the old database.

In PostgreSQL: You use native logical replication (CREATE PUBLICATION on the new database and CREATE subscription on the old database). Before you switch traffic to the new cloud, you promote the new database to primary, pause writes, record the Log Sequence Number (LSN), and reverse the replication direction so the old database becomes the subscriber.
In MySQL: You configure binary log (binlog) replication with Global Transaction Identifiers (GTIDs) enabled on both sides. When the new database takes over as primary, you point the old database's replication channel (CHANGE REPLICATION SOURCE TO) at the new primary so it pulls every new transaction back to the old cloud.

Verify that primary keys and auto-increment sequences do not collide, and ensure your network firewall rules allow encrypted database traffic to flow in both directions between the two clouds.

4. Sync new object storage writes back to the old bucket

If your application stores user uploads, invoices, or media files in object storage (like AWS S3, Google Cloud Storage, or Azure Blob Storage) and you are moving to an S3-compatible bucket on an EU provider, you need a plan for new files created after cutover.

You have two practical options:

Continuous reverse sync: Run a background worker or scheduled job (using tools like rclone with checksum verification) every few minutes to copy newly created or modified objects from the new EU bucket back to the old cloud bucket.
Application dual-writing: Update your storage adapter code to write new files to both buckets simultaneously during the 30-day transition period, while reading from the primary cloud.

Avoid syncing deletions automatically to the old bucket during the rollback window. If a bug in the new environment accidentally deletes objects, you want those files to remain intact in the old bucket.

5. Enforce a clean write pause during the switch

Whether you are migrating forward or rolling back, never switch active database primaries while both sides are accepting writes. Doing so causes split-brain data conflicts that take days to untangle manually.

Build a simple "maintenance mode" or read-only toggle into your application or load balancer. When executing the cutover—or when executing a rollback—follow this exact sequence:

Enable read-only / maintenance mode to stop incoming writes.
Terminate any remaining active write connections on the current database.
Wait until replication lag between the two clouds reaches zero (typically under 5 seconds on a healthy link).
Promote the destination database and reverse the replication stream.
Switch DNS or load balancer targets and disable maintenance mode.

Defining your rollback triggers and decision matrix

The worst time to decide what constitutes a "failed migration" is at 2:00 AM when half the team wants to debug a broken service and the other half wants to abort. Human psychology works against you during a migration: because the team invested weeks of work into preparing the move, engineers naturally suffer from sunk-cost bias and will try to fix problems forward for too long.

Protect your team by writing down hard, measurable rollback triggers before the migration starts. Designate one person as the Incident Commander who has the authority to call a rollback when a trigger is breached.

Failure Scenario	Metric / Trigger Threshold	Action Required	Phase
Replication lag won't clear	Lag remains > 5 seconds during write pause for more than 10 minutes	Abort cutover; re-enable writes on old cloud	Phase 1 (During cutover)
Smoke tests fail	Critical path (login, checkout, API auth) fails for > 15 minutes after switch	Abort cutover; switch DNS back to old cloud	Phase 1 (During cutover)
Elevated HTTP 5xx errors	5xx error rate exceeds 2% for 15 consecutive minutes after fixing attempts	Execute stateful rollback via reverse replication	Phase 2 (Post-cutover)
Database saturation	CPU or disk I/O stays above 90% under normal load and scaling up fails	Execute stateful rollback during next low-traffic window	Phase 2 (Post-cutover)
Data corruption or loss	Any verified instance of lost transactions or mismatched checksums	Immediate emergency pause and rollback evaluation	Phase 1 or Phase 2

Regulatory context: DORA and the EU Data Act

For many European companies, having a tested rollback and exit plan is no longer just an engineering best practice—it is a legal requirement or a contractual obligation to enterprise customers.

Since 17 January 2025, the Digital Operational Resilience Act (DORA, Regulation (EU) 2022/2554) has applied to financial entities across the EU and their critical third-party ICT service providers. DORA requires covered organizations to maintain comprehensive ICT business continuity policies and documented exit strategies for critical functions. Crucially, regulators expect exit plans and migration procedures to be tested and to account for failed transitions without disrupting service delivery or losing data integrity.

At the same time, the EU Data Act (Regulation (EU) 2023/2854) makes keeping a rollback option financially easier. Historically, cloud providers charged high data egress fees whenever you replicated data out of their network. Under Article 29 of the EU Data Act, cloud switching charges—including data egress fees incurred for switching providers—are restricted to no more than the provider's direct cost until 12 January 2027, and are completely banned from 12 January 2027 onward. While you will still pay for the compute and storage resources while running two environments in parallel for a month, punitive network transfer fees for syncing data between providers are phasing out.

Pre-migration rollback checklist

Before you approve your migration window, verify every item on this list:

[ ] Old environment freeze: Automatic teardown scripts, billing cleanup jobs, and lifecycle deletion rules on the source cloud are disabled for at least 30 days.
[ ] DNS readiness: TTLs on all production domains and subdomains have been lowered to 60–300 seconds at least 48 hours prior.
[ ] Access verification: At least two engineers on the cutover call have working admin credentials and MFA access to both the old cloud console, the new cloud console, and the DNS registrar.
[ ] Write-pause mechanism: A tested script or feature flag exists to block all user and background worker writes within 30 seconds.
[ ] Reverse replication tested: Logical replication from the new database back to the old database has been tested in a staging rehearsal.
[ ] Object storage sync active: A script or worker is ready to sync new S3/object storage files from the new cloud back to the old cloud.
[ ] Timebox agreed: The team has agreed on a strict maximum debugging time (for example, 20 minutes during the cutover window) before triggering an automatic rollback.

Frequently asked questions

How long should we keep our old cloud environment running after migrating?

For most production workloads, keeping the old environment running in a synced standby state for 14 to 30 days is the sweet spot. Two weeks covers weekly batch jobs and normal traffic cycles, while 30 days ensures you survive a full month-end billing or reporting cycle on the new infrastructure before deleting the old resources.

Does keeping a rollback environment double our cloud bill?

It temporarily increases your infrastructure spend for the duration of the overlap, typically one month. However, because European cloud providers like Hetzner, UpCloud, Scaleway, and OVHcloud generally have lower baseline compute and bandwidth prices than US hyperscalers, running the new EU environment for a month alongside your old cloud usually pays for itself within a few months of shutting the old provider down. You can also scale down non-essential worker nodes in the old cloud while keeping the database warm.

Can we roll back if we change database engines during the migration?

Rolling back across different database engines (for example, moving from AWS DynamoDB to PostgreSQL, or changing major database versions) is much harder because native logical replication requires compatible engines and schemas. Whenever possible, keep the exact same database engine and version during your cloud migration, and save database modernization projects for after you are stable on the new cloud.

What should we do with background queues during a cutover or rollback?

Before pausing database writes, pause your queue consumers (such as Sidekiq, Celery, BullMQ, or RabbitMQ workers) and let in-flight jobs finish. Wait for the queues to drain completely before switching environments, or ensure unprocessed job payloads are replicated to the destination Redis or queue instance so jobs are not dropped or executed twice.

Sources

Drafted with AI assistance and fact-checked by the MoveInfra team. The original, kept up to date, is at moveinfra.eu. Free tools for pricing a move to an EU cloud: moveinfra.eu/tools.

DEV Community