Each of our depots has one or two network switches in a cabinet, feeding the wall sockets in the offices, the wireless access points in the warehouse and yard, the label printers, the tills on the trade counter and the desk phones. They were installed by a contractor in 2018 to a standard design, and have been reliable enough that I rarely think about them.
On a Tuesday in May, at about half past ten, everything at one of our Yorkshire depots stopped at once. The scanners dropped their connections, the tills froze, the phones went silent, and our monitoring lost sight of both switches. From head office we could reach nothing on the site.
The cause was a loop. That morning the transport office had been rearranged, and someone tidying the cables under a desk found a patch lead with one end hanging loose and plugged it into the spare socket beside it. Both ends of the lead were now in wall sockets connected to the same switch. An Ethernet frame carries no count of how many times it has been forwarded, so a broadcast sent into a loop goes round indefinitely, and every pass through a switch copies it again. Within seconds both switches were spending all their effort on the same few messages circling, and had nothing left for anything else, including answering us.
Switches are built to stop exactly this. The spanning tree protocol notices when two ports lead to the same place and blocks one of them, and most switches can also limit how much broadcast traffic a port may send. Ours could do both. When we finally read a copy of their configuration, spanning tree had been switched off across the estate when the switches were installed. It seems that in its default form it holds a newly connected port for about thirty seconds before passing traffic, and somebody had complained that the tills were slow to connect in the morning.
It took nearly four hours to find, because the only way to find a loop on a switch you cannot reach is to unplug things. Our engineer, an hour and a half away, talked the depot manager through it on her mobile, one cable at a time, until the lights on the switch calmed down. The depot lost most of a morning's picking and forty three deliveries went out late.
That week we checked the rest of the estate. Twenty six of our thirty one depots had the same configuration. All of them now run spanning tree, with the sockets that face users set to connect immediately but to shut themselves down if a loop or another switch appears behind them. Storm control limits the broadcast traffic any one port can send. Wall sockets that are not in use are disabled at the switch, and bringing one into service is a ticket. And each depot now has a separate management connection to its switches over a mobile router, so we can see inside a site when its own network has failed.
A network should not be one stray cable away from stopping a business. Design it to survive the ordinary mistakes that people in a busy office will make, because they will make them, and they have no way of knowing which ones matter.
– Serguey Shinder
Top comments (0)