I recently moved a production backend service from one cloud provider to another. Solo. Zero downtime. Real users, real traffic, no maintenance window.
The real reason I moved it: auto-scaling and spot instances cheaper, flexible compute that scales up and down with demand never worked cleanly on the old provider. I kept fighting it instead of benefiting from it. On top of that, I kept running into permission issues that had nothing to do with what I was actually trying to do. Both were signals the same thing: time to move to a cleaner setup, on infrastructure we already had access to, where I could design things properly instead of working around old debt.
The actual migration took two days. Getting the architecture right took a month.
I used to think migration was the hard part. It's not. Design is.
Here's what that month actually looked like including the parts I got wrong before I got them right.
The wrong turn I almost shipped
The old system had a setup step: each machine needed to establish its own identity before it could do real work. The old approach was simple configure one machine correctly, capture its state, copy that state onto every new machine. It had worked fine there for a long time.
I assumed the same approach would work on the new provider. It didn't. Figuring out why took three tries.
Try 1: captured the state, copied it to new machines. Every one failed the same way.
I traced it back to how the original setup was done through a remote session, by clicking an icon, instead of launching the software with a flag that tells it to use an isolated local config. Without that flag, the setup silently saved itself somewhere else entirely. The "source" machine was never actually configured right in the first place.
Try 2: fixed that, redid the setup, recaptured, recopied. Same failure, on a different fresh machine, even though I'd verified the source was correct this time.
That ruled out my first theory.
Try 3: tried a literal clone of the working machine, same hardware profile, no changes. Same failure. Ruled out hardware identity too.
I also tried automating the manual setup step with a scripting tool, so no human would ever need to do it. It could find the right window real progress but couldn't actually type into it. A session can exist without a real display attached, and that turns out to matter.
The actual fix: stop copying state entirely. Install fresh, log in through a plain config file, let each machine do its own setup from scratch. No copying, no cloning. It worked immediately, on the very next machine, with zero manual steps.
The lesson: don't copy identity between machines. Let each one earn its own.
The number I didn't expect
Before rolling this out for real, I ran an actual load test real concurrent usage, not a guess.
The new machine had the same CPU as the old one, just more memory. I assumed load would scale roughly the way it had before. It didn't. At light load, CPU sat around 22%. At moderate load, it jumped to 92% nowhere near a straight line.
I lowered the safe capacity limit by about half, based on that real number, instead of keeping the number that had worked on the old setup. More memory didn't matter here the real limit was CPU, and the only way to find that was to actually test it.
The same test also caught a real bug: one failed connection was silently blocking the next unrelated one, because two different failure types looked identical to my code. Only showed up under real concurrent load.
A trade-off I made on purpose
Once more than one machine was handling traffic behind a shared entry point, a returning user could land on a different machine than before. Fixing that properly meant building a much more complex routing layer.
I chose not to build it. The system already had a way to recover gracefully when that happened it just set itself up fresh wherever it landed. Building the complex fix would have solved a problem the system already tolerated fine. I wrote the trade-off down clearly, including exactly when I'd need to revisit it.
Choosing the entry point on purpose
Two reasonable options existed for routing traffic in: a simple passthrough, or a heavier one that handles encryption itself. The service already handled its own encryption. Picking the heavier option would've meant solving a problem I didn't have. I went with the simple passthrough.
I also rolled it out in two small, safe steps instead of one big one: first, add the new entry point alongside the old direct access, so I could prove it worked with zero risk. Only once that was confirmed did I remove the old path. Two small reversible steps beat one big risky one.
Scaling the actual reason I moved
This was the part I was most careful about, since it's what drove the whole move.
The setup could scale a small always-on baseline, with cheaper burst capacity added on top when needed. That part was straightforward.
What I deliberately didn't do yet: build a fully automatic policy that scales on its own based on load. I didn't have real usage data yet, and an automatic policy built on a guess is worse than no policy it either reacts too late or too early, and you won't know which until it's live.
Instead, I scaled manually for the specific period it mattered, watching real alerts. That gave me real numbers under real load. Only after that did building a real automatic policy become a genuine next step instead of a guess.
Designing access properly this time
Since permission friction was part of what pushed this move in the first place, I made sure not to carry the same problem forward. I reviewed every broad permission the new setup had, kept the ones that genuinely couldn't be scoped any narrower, and added explicit restrictions on the sensitive actions that could be. That review caught one real gap before it caused a problem a resource was missing a tag that a new safety rule depended on, which would have accidentally blocked a legitimate action.
The principle: design access narrowly from day one. It's much harder to tighten permissions later than to grant them as they're actually needed.
Automating the pipeline, deliberately
I set up the deployment pipeline to be fully automatic any change that passes its checks deploys on its own, no manual approval step in between.
This was a deliberate choice, not laziness. If every deploy needs someone to manually approve it, that someone becomes a bottleneck and in practice, that someone is me, getting pulled in for routine changes that don't actually need a human gate. Automating it means any developer on the team can ship their own change without needing to loop me in every time. Fewer interruptions, faster shipping, same real safety controls just enforced differently through proper checks and scoped permissions, not a manual click.
Alerts for everything that can fail
Simple rule I followed throughout: if something can fail silently, it needs an alert. Not one flat "something's wrong" notification different problems need different urgency. A minor slowdown shouldn't page the same way a real outage does.
Getting this right early meant I could rely on watching alerts instead of guessing, during the manual-scaling window above and afterward.
Retiring the old system carefully
Once the new setup was proven, I didn't just delete the old one. Before removing anything, I checked what was still actually depending on it and found one thing still quietly running against it that I would have broken. Instead of accepting that disruption, I fixed the actual gap first, confirmed nothing else was still depending on the old system, and only then removed it for good.
The takeaway
None of these decisions were obvious going in. Each one meant testing an assumption instead of trusting it that state would copy over cleanly, that load would scale the way it used to, that an old permission setup was fine to carry forward.
The month wasn't spent writing documents. It was spent finding out which assumptions were wrong before they became real problems. That's what made the two-day execution possible not extra planning for its own sake, but planning that had already been tested against reality.
Curious if others have hit the same wall with auto-scaling or spot instances on a provider that just doesn't handle it cleanly feels like a more common trigger for switching providers than people usually admit.
Top comments (0)