Our change policy still names a standard maintenance window: Sunday, two until six in the morning. It was written when the company traded in one country and shut at six on a Friday, and for about a decade it was an accurate description of when nothing was happening.
Nothing about it is accurate now. The warehouse runs a night shift. We have customers in two other time zones who regard Sunday as a working day. A partner integration polls us hourly and does not observe our calendar, and the month end close, which the finance team performs at whatever hour it takes, has twice landed squarely in the window. Last year we made just under nine hundred production changes and fewer than sixty of them happened inside it.
The window did not disappear cleanly, which would have been easier. It survived as a formality, and the formalities caused their own damage. Anything genuinely disruptive queued for the few Sundays that were still defensible, so three teams would land significant work in the same four hours, and when one of them had to roll back the others were rolling back at the same time with one engineer on call. The riskiest changes we make were therefore performed by tired people at three in the morning, which is not a control, it is the opposite of one. And the work that could not find a slot simply slipped. Two of our firmware updates were eighteen months overdue for no reason other than diary arithmetic.
We have stopped treating availability as a property of the estate and started treating it as a property of each service. Every significant system now has an agreed availability statement from its business owner, in writing, saying when it may be interrupted and for how long, and several of those turned out to be far more generous than we had assumed. Where a service genuinely cannot stop, we spend the money on making changes possible without stopping it, and only for those. A slot holds one significant change and is booked.
The quiet hour was a fact about the business, not about technology. When the business changed, we kept scheduling against a fact that had expired.
– Serguey Shinder
Top comments (0)