Two developers on the billing team each wrote a database migration in the same week. The first one, adding a column for invoice language, became V41. The second one, an index, became V42. The index was merged and deployed on Monday. The invoice language change was merged on Tuesday and deployed that afternoon. Sixteen days later, the first customer to pick a language for their invoices got an error page.
The column did not exist in production. Our migration tool, Flyway, records each migration it applies and normally refuses to start if it finds one in the code with a lower version than the highest it has already applied, because that usually means history has been rewritten. Two years earlier, during a hotfix that needed exactly that situation to pass, somebody had set the tool to ignore such migrations instead. The setting was meant to be temporary. With it in place, production saw V41 after V42, decided it was simply in the past, and skipped it without a word. The application started, every health check passed and no other code touched the column.
Staging had the column. Staging is rebuilt from scratch every night, so it ran V41 and V42 in the right order and never encountered the problem.
We applied V41 by hand that morning. The ignore setting is gone, so a migration arriving out of order now stops the deploy at startup, loudly, in the first environment that sees it. Migration versions are no longer sequence numbers that two branches can both claim. They are timestamps taken when the file is created, and a check in the merge queue fails any pull request whose new migration is older than the newest one already on main. And after every production deploy, a job compares the list of applied migrations in production with the list in the code and alerts on any difference. Its first run found a migration from 2023 that had been skipped the same way.
A tool that refuses to start is easy to argue with during a hotfix. The refusal was the only part of the system that understood order, and we had switched it off once and then forgotten that it was off.
– Sergey Shinder
Top comments (0)