On the ninth we switched the delivery preferences read path onto the new column. The deploy took four minutes and went perfectly. Twenty minutes later support had a hundred and forty tickets from customers whose preferences page was blank, and a fair number of them had selected something in it that week.
This was a normal three step change. Write both formats, backfill the old rows, then read from the new column. Step two was a script that walks forty one million rows in batches of five thousand, and it takes about four days at a rate the database is happy with. Nobody wanted a four day job in a pipeline, so it was started by hand from a jump host, inside a screen session, on the second of the month.
On the Sunday our own patching automation rebooted that host. The session went with it. The script had no checkpoint table, reported progress by printing to standard output, and left nothing behind that said where it had got to. It had covered sixty three percent of the rows. The engineer who started it was on leave the following week, and the ticket said backfill complete, because he had run it and it had been running when he last looked.
Long jobs are now Kubernetes Jobs. They record a checkpoint row per batch and resume from it, so a restart costs one batch rather than everything. They publish rows remaining as a metric, with a dashboard, and the alert is on no progress for fifteen minutes rather than on failure, because this one never failed. The switch itself is now guarded: the deploy runs a count of rows still in the old shape and refuses to proceed while it is not zero. That guard is four lines and would have been enough on its own.
A release that takes four minutes can depend on work that takes a week, and your pipeline has nothing to say about the week. It will tell you the deploy succeeded, which was true, and it has no opinion at all about whether the thing you deployed was ready to be switched on. Anything a release waits for has to be able to say how far it has got.
– Sergey Shinder
Top comments (0)