We shipped at half past nine on a Tuesday and for the next eleven minutes roughly one order write in three failed with a database error about an unknown column. It stopped on its own. By the time the second person had joined the call it was already recovering, which is its own kind of confusing.
The release dropped a column called legacy_status. Nothing in the new code touched it. The drop had been reviewed, it was the tidy end of a piece of work that had taken two months, and it ran as a pre deploy migration because that is the order our pipeline has always used and the order that is right for adding things.
Then the rolling update started. Thirty pods, replaced a few at a time under a disruption budget, taking eleven minutes to complete, which is exactly what we had asked for and normally exactly what we want. For all eleven of those minutes, some fraction of the fleet was running the previous version, and the previous version builds its insert statement from its own mapping of the table, listing every column it knows about. Every write from an old pod named a column the database no longer had.
The rollback made it plainer. Rolling the deployment back would have restored code that requires the column, against a database that no longer has it, and the contents are not coming back from anywhere except a restore. For those eleven minutes we had no version of the application we could safely run.
The rule now has teeth rather than being a paragraph in a wiki page. A migration is classified as additive or destructive by a check that reads it, and a release may not contain a destructive migration together with application changes. Dropping something is its own release, at least one release after the code stopped referring to it, run behind a manual gate in the quiet window. Every migration states how to reverse it, and a destructive one that cannot be reversed says so out loud in the approval.
During a rolling deploy there is no single version of your application. There are two, for as long as the rollout takes, and the schema has to be one that both of them can live in.
– Sergey Shinder
Top comments (0)