Most of our systems now run in the cloud, but some still live in the server room at head office: the database servers behind our warehouse system and the storage behind our file and print servers. When something in that room needs hands rather than a remote session, the on call engineer drives in. Our runbook says so in two words: attend site.
At ten past two on a Sunday morning in January a storage controller failed and did not hand over cleanly to its partner. The warehouse database stopped. Our first depot shift starts at six, and without that database nothing can be picked. Our engineer saw the alert, tried what he could from home, and was in the car park by quarter to three.
His card did not open the door. In the autumn our facilities team had replaced the building's access system, and to cut false alarms and the cost of night callouts they had limited entry between ten at night and six in the morning to a list of keyholders. It was a sensible change. The list held four names: two facilities managers, the office manager and the managing director. Nobody in IT was on it, and nobody in IT had been asked.
He rang the alarm monitoring company, which could only contact a keyholder. The first facilities manager was on holiday. The second answered at twenty past four and lived fifty minutes away. Our engineer sat in his car for two hours and twenty minutes with a fault he cleared in about twenty once he reached it. The database came back at ten to six, the depots started picking late, and the first deliveries left about an hour and a half behind.
What stung was that every part of our out of hours arrangement had been tested except this one. We had rehearsed failovers, restored backups and walked through the call tree. We had never sent an engineer to the building in the middle of the night to see whether he could get in, because getting in had always been taken for granted.
We changed four things. Every engineer on the on call rota is now on the keyholder list, and the rota goes to facilities each month so the list follows it. The server room has a second way in that does not depend on the access system, a key in a safe whose code is held with the on call phone. Changes to building systems now appear on the same change calendar as ours, so facilities and IT see each other's plans before they go live. And once a quarter, on a night chosen without warning, the on call engineer drives to head office and opens the server room door, and the time it took goes into the report beside our restore tests.
Recovery plans are written about systems, and every one of them quietly assumes that somebody can stand next to the machine. Test the journey to it, door included, as seriously as you test the machine itself.
– Serguey Shinder
Top comments (0)