In the third week of July three of our depots lost their network within four days of each other. In each case the switches had shut themselves down to protect themselves from heat. One cabinet sits on a mezzanine directly under a metal roof, where we later measured forty six degrees at three in the afternoon. One is in what used to be a cleaner's cupboard, beside a hot water cylinder. The third is above the canteen kitchen.
None of those rooms was chosen for the equipment. When each site was fitted out, somebody asked where there was space for a cabinet, and the answer was whichever room nobody else wanted. That worked for years because the summers allowed it. The last four summers have not, and we had not noticed the pattern because each outage had been logged as a hardware fault at a single site.
The awkward part is that the equipment had been telling us. Every switch we own reports its internal temperature. We had never collected it, because the monitoring was set up to ask whether a device was alive, not whether it was comfortable.
We now collect temperature from every switch and access controller, which cost nothing but an afternoon of configuration, and alert at a threshold well below the shutdown point. Looking at last summer's figures, nine of our thirty nine sites had spent days above that threshold. We surveyed every room that holds equipment and scored it for heat sources, ventilation, dust and water. Two cabinets have been moved, four rooms have extraction fans, and the mezzanine has a small cooling unit that facilities now maintain on the same schedule as the canteen's. The standard site kit specifies the room as well as the equipment in it, and the site opening checklist asks where the cabinet will go before anybody orders it.
Most of the equipment had never been faulty. Two depots that used to replace a switch every year or so have not needed to since.
We picked those rooms by asking where there was space. The question that mattered was where the heat goes in August, and the buildings had been answering it every summer.
– Serguey Shinder
Top comments (0)