Most of our systems have moved to the cloud, but the server room at head office still holds the phone system, the core of the network that every depot's traffic passes through, and a few servers we have not yet retired. It is protected by an inert gas suppression system installed in 2014. If two detectors see smoke, it floods the room and puts out a fire without water reaching the equipment.
In 2023 a contractor sanding the corridor outside set off the detectors and the system discharged. Refilling the cylinders cost fourteen thousand pounds. To stop it happening again while the work went on, the facilities manager switched the panel to manual release only. The work finished. The panel stayed as it was, and that facilities manager left the following spring.
The panel hangs in the corridor, and I walked past it most days. It had a small amber light marked automatic off. I had never asked what it meant. The suppression system belonged to facilities, its annual service was on their contract, and each year the engineer's report said, accurately, system left in manual mode at client request.
In August a string of batteries in one of our UPS units overheated and began to smoke. The detectors raised the alarm and the building was evacuated. Nothing was released. The fire service arrived eleven minutes later, cut the power to the room and dealt with the batteries with extinguishers. The damage was limited to one cabinet, but the room was dark for five hours, and with it went the phones and the network to thirty nine depots.
Nobody had done anything unreasonable. A sensible temporary decision outlived the person who made it, and the only people who could see its effect were the ones who did not know what it protected.
The panel's state is now wired into our monitoring through a spare relay output, and anything other than automatic for more than a day raises an alert to my team and to facilities. Any work on fire, power or cooling for an IT room goes through our change process as well as theirs. The service reports now come to us too. UPS batteries are replaced by age, at five years, and their temperature is monitored.
A protection system switched to manual looks exactly like one that is working, right up until the moment it is needed.
– Serguey Shinder
Top comments (0)