Our rollback was one command and took ninety seconds. We were proud of it. Then we shipped a release with a schema migration, hit an unrelated bug, ran the rollback, and watched the old version crash on startup against a database it no longer understood. The application went back. The schema did not. We were down for another forty minutes while someone hand-wrote SQL under pressure.
The migration had dropped a column the new code stopped using. Perfectly clean change, reviewed and approved. But the previous version still selected that column in a query, so the instant we reverted the code, every request hit a missing-column error. Rolling back the schema meant restoring data we'd already deleted, which is not a rollback, it's a restore, and it needed a backup and a maintenance window we hadn't planned for.
That was the day I learned that code rollbacks are cheap and schema rollbacks usually aren't. Code is stateless — put the old artifact back and it's the old system. A database has state, and destructive migrations throw information away. You cannot un-drop a column any more than you can un-send an email.
So we adopted expand-and-contract and never left it. A change that used to be one migration becomes a sequence across releases. Expand: add the new column, keep the old one, deploy code that writes both and reads the old. Then a release where code reads the new column. Then, only after the previous version is well and truly out of production and you're confident you won't revert past it, contract: drop the old column in a separate release with no code change in it. Every individual step is backward compatible with the version before it, which means every step is rollback-safe.
It's more releases and more discipline, and people push back on the ceremony. But the property you buy is exact: at any point in the sequence, the previous deployment still runs against the current schema. Rollback stays a ninety-second command instead of an incident inside an incident.
Your rollback is only as fast as your least reversible change. If a release contains a destructive migration, you don't have a rollback plan. You have a recovery plan, and you should know which one you're holding before you press the button.
– Sergey Shinder
Top comments (0)