Every Windows computer we own runs CrowdStrike's Falcon agent, which protects it against malware. When we chose it, I made a point of the update policy. New versions of the agent reach a test group first, and the rest of the estate stays one version behind the latest, so nothing new lands on a till or a goods in PC before we have lived with it. I described this to our audit committee as control over change.
On Friday 19 July 2024, at nine minutes past five in the morning UK time, CrowdStrike released an update that crashed Windows machines around the world, about eight and a half million of them by Microsoft's estimate. It was withdrawn seventy eight minutes later. Our PCs were not protected by being one version behind, because it was not a version. It was a content file, a set of instructions about what to look for, which the agent downloads whatever version it is running. Our policy governed the agent's software. It had never governed this, and I had not known the difference existed.
Which of our machines were hit came down to which were switched on in those seventy eight minutes. Most branch PCs were still off. Our distribution yard works through the night, and its goods in and despatch PCs went to a blue screen, along with eleven servers, including the one that prints delivery notes. Each machine needed someone at the keyboard to start it in recovery mode and delete one file. The first lorries left three hours late, and the last server was back on Saturday morning.
CrowdStrike has since given customers a way to stage content updates as well, and we use it: our own team and two branches first, everyone else some hours later. The larger change was in how I think about it. We listed every product that changes itself on our machines, from the security agent and the browser to printer firmware and the trading system's client, and for each one asked which channels it updates through and which of those we actually control. Several had a second channel we had never noticed.
A setting that controls versions can leave you believing you control change. Ask what else the product downloads, and who decides when.
– Serguey Shinder
Top comments (0)