Friday, ten past four. A release went out, error rates climbed inside two minutes, and we made the easy decision to roll back. The rollback failed after eleven seconds with a manifest unknown error. The image we had been running all week did not exist any more.
Our registry had a lifecycle policy keeping the ten most recent images per repository. It was written at a time when we merged a few times a day, when ten images meant a fortnight of history. Since then we had added a nightly dependency rebuild, per branch preview builds pushing to the same repository, and a build on every merge to main. Ten images now covered about thirty hours. The version that had been serving production since Monday was deleted on Wednesday night by a rule nobody had opened in two years.
So we rolled forward instead. A patch, a review from someone who had already started the weekend, and fifty minutes of degraded service in place of four.
What changed. The deploy pipeline applies an immutable tag to any image it promotes to production, and the lifecycle rule excludes that tag, with a separate rule keeping anything ever deployed for a year. Preview and branch images now expire after seven days, which is where the storage bill actually came from; the images we were so carefully deleting cost about eleven dollars a month to keep.
More useful than either of those: a scheduled job asks, for every service, whether the currently deployed image and the one before it are still present and still pullable, and fails loudly if either is not. It is a rollback rehearsal that does not require a rollback. Switching it on found two more services whose previous version had already been collected, and one where the chart values for the previous release had never been stored anywhere at all.
A rollback plan is a claim that something still exists. Ours was being quietly deleted every night by a policy written for a repository that no longer resembled the one we had.
– Sergey Shinder
Top comments (0)