In January 2024 I took over the backend of WeWALK, a smart cane platform for visually impaired people. The codebase dated back to 2020 and had passed through a few hands before mine - and underneath it was a live production system with real users depending on it.
27,000 registered users, to be exact - and I want to be honest about that number: it is not impressive. People migrate databases with millions of users and write nothing about it. What made this migration interesting was not how many users there were. It was three other things:
- The users cannot be interrupted. For many of them the app is part of how they get around a city. "Please update to continue" is a minor annoyance in a food delivery app; in an accessibility product it can mean someone's cane stops talking to their phone while they are out somewhere. So one rule was fixed before any planning: nobody gets forced to update. Every client binary in the field keeps working.
- The data never stops arriving. These are not users who open an app twice a day. Every active cane sends a 102-byte diagnostic snapshot roughly every ten seconds, flowing into a partitioned time-series table. Multiply by a fleet of devices and you have a write stream that never pauses - including during your migration. There is no maintenance window when the "users" are canes on the street.
- The data was heterogeneous. MongoDB and PostgreSQL on the old cluster, Firestore entering the picture on the new one, plus IAP billing events, a rules-based entitlements engine, and an audit trail doing JSONB before/after diffs. This was not one big table to copy. It was several kinds of data with different consistency needs, all moving at once.
Oh, and one more thing: the old cluster was effectively a black box.
Starting from backups
The stack ran on a self-hosted Rancher/Kubernetes cluster. By the time the migration started, working access to that environment was no longer practical - the kind of thing that happens naturally over years of team and vendor transitions, and more common in the industry than anyone likes to admit. What we did have, reliably, was backups.
So the plan became: reconstruct what the system actually was from its backups, and bring that up on Google Cloud Run, Cloud SQL and Firestore instead of trying to revive the old machines.
If you have never rebuilt a system you cannot inspect, I recommend it as an educational experience and nothing else. You find out exactly which parts of the configuration were written down and which parts had never made it into documentation anywhere. Every service that came up was a small negotiation between what the backup said, what the code expected, and what production traffic proved.
Old endpoints, new cloud
Part of the platform lived on AWS, and the mobile apps talked to WebSocket endpoints there. Those addresses are baked into client binaries in the field. Moving to GCP the normal way means new endpoints, which means a mandatory update - the one thing we had ruled out.
So the old endpoints simply never died. I built a Lambda-to-Cloud-Run proxy that keeps the existing AWS WebSocket addresses alive and forwards everything to the new infrastructure. The oldest client build in the field talks to the same address it always did; it just has no idea the thing answering it moved. Nobody updated anything. Support tickets from the migration: none that I know of.
The most valuable tool was a diff
Later we moved nine services between regions, europe-west3 to europe-west1. The thing that saved us was not clever. Before cutover I diffed the configuration of every service between source and target - every environment variable, every setting, service by service.
That diff caught four missing environment variables. Four. Any one of them would have been a production incident discovered at the worst possible time. After years of running both Kubernetes clusters and serverless platforms, my honest conclusion is that migrations rarely fail in interesting ways. They fail because an env var didn't come along. Diff first. It is boring and it works.
Moving data that will not sit still
The data layer had its own migration, Firestore to PostgreSQL - executed while the telemetry stream kept writing. We did it as a dual-write: write to both stores, keep reading from the old one, and put the read cutover behind an environment flag. When the new store had proven itself under the live write load, we flipped reads. The old path stayed in place as an escape hatch, with graceful degradation if anything went sideways.
The time-series side needed its own care: continuous device snapshots do not tolerate a "stop the world, copy, restart" approach, which is exactly why the table is partitioned and why the cutover had to be a flag flip rather than a batch job.
That schema has since grown from 34 tables to about 95, across 174 migrations. The system that had to be rebuilt from its backups in early 2024 is now the boring, observable, redeployable kind of production system I like to run.
What stayed with me
The "no forced updates" constraint felt like a burden and turned out to be a filter. It eliminated every lazy option early, and what survived was genuinely better architecture. And the two least glamorous artifacts of the whole project - a pile of backups and a configuration diff - did more for those users than anything clever I wrote.
If you are inheriting an undocumented production system: test whether you can rebuild from your backups before you need to, and diff your configuration before every cutover. That is the whole post, really.
Top comments (0)