Our main distribution centre runs its warehouse system on servers hosted by the software supplier, reached over a private network line that a second supplier manages for us. Both have maintenance windows written into their contracts, and both use them.
In March, on the same Sunday night, the warehouse supplier upgraded the database underneath our system and the network supplier replaced the router at their end of our line. Each had told us. The warehouse supplier emailed our application team three weeks earlier. The network supplier posted a notice on its customer portal, which our infrastructure team only opened when something went wrong. Neither notice reached our change calendar, because the calendar only held changes my own team was making.
At six on Monday morning the day shift took over and the handhelds could not stay connected. The warehouse supplier's engineers saw a healthy database and a network that dropped connections after a minute. The network supplier saw a working line and an application that kept opening connections and abandoning them. For five hours each sent us evidence that the fault was the other's. It was a setting on the new router that closed idle connections sooner than the upgraded database expected, and once we put both suppliers on one call, chaired by us, they found it in forty minutes. By then the day shift had picked about a third of its normal volume, and ninety one customer deliveries went out late.
Nobody had done anything wrong on their own terms. Each change was planned, approved and announced. What was missing was one place where someone could see two changes landing on the same service on the same night and ask whether both should.
Our change calendar now holds every change that touches our services, whoever makes it. Each supplier contract names who in my team receives their notices, and that person enters them. When two changes affect one service in the same window, one of them moves, and the supplier who moves is told why. Since April the calendar has caught six clashes, two of them between suppliers who did not know the other existed.
A change calendar that only lists your own work shows you the changes you control. The changes your suppliers make to your services can break them just as easily, and they belong on the same page.
– Serguey Shinder
Top comments (0)