DEV Community

Cover image for One bad config file, 8.5 million blue screens
Kashif Manzer
Kashif Manzer

Posted on

One bad config file, 8.5 million blue screens

#devops #security #programming #beginners

Friday, 19 July 2024. Early morning in Europe, late night in the US.

At airports, departure boards froze mid-update. In hospitals, staff reached for pen and paper. At banks and broadcasters, screens everywhere showed the same thing: a plain blue screen, a frown, and the words "Your PC ran into a problem."

It was not a cyberattack. Nobody had been hacked. Something had been updated, and the update was poison.

The poison

The poison came from CrowdStrike, a security company whose Falcon software runs on millions of corporate Windows machines. Falcon watches for attacks, and to keep up with new ones, CrowdStrike feeds it fresh detection rules constantly. Not once a month. Sometimes several times a day.

These rapid updates are called Rapid Response Content. They travel through numbered channel files. On 19 July at 04:09 UTC, Channel File 291 went out with new rules for spotting attacks that abuse named pipes. (A named pipe is just a way for two programs on the same machine to talk to each other. Attack tools love them, so defenders watch them.)

78 minutes later, at 05:27 UTC, CrowdStrike pulled the file back. But any machine that had grabbed it in that window was already doomed, and the doom had a special cruelty to it. The bad file loaded during boot. So the machine would start, load the file, crash, reboot, and start again. Forever. Fixing it meant visiting each machine by hand, booting into Safe Mode, and deleting the file.

About 8.5 million Windows machines, by Microsoft's count.

Where the poison lived

To understand how one bad file could do this, you need to know where Falcon lives. It does not run like a normal app. Its core, a driver called CSAgent.sys, runs in kernel mode.

A quick explainer, because this is the whole story. Your computer has two neighbourhoods. User mode is where apps live. If an app crashes there, the app dies and the computer carries on. Kernel mode is where the operating system core lives. If code crashes there, the whole machine goes down with it. Security software runs in kernel mode because it needs to see everything. The price is that its bugs become everyone's bugs.

Now the actual defect. Months earlier, CrowdStrike had added a new detection capability: an "IPC Template Type" for watching named pipes, shipped in sensor version 7.11 on 28 February 2024. Think of a template type as a form with blanks. It declares the inputs a detection rule is allowed to use. This one declared 21 input fields.

The sensor code that reads those inputs, the Content Interpreter, had been built to handle 20.

On 19 July, Channel File 291 carried the first rule that actually filled in field number 21. Earlier rules had left that field on a wildcard match, which is why months of testing and four earlier deployments in March and April never tripped the bug. The moment real data landed in field 21, the interpreter read past the end of its list. In plain terms: it reached for item 21 in a list of 20.

In user mode, that is one crashed program. In kernel mode, it is a blue screen of death. And because the file loads at boot, the blue screen arrived before anyone could do anything about it.

A sketch of the shape of the bug (illustration only, not CrowdStrike's real code):

# the template type declares 21 fields
fields = read_template_instance()  # 21 items this time

# but the interpreter was built for 20
for i in range(21):
    value = field_list[i]  # i = 20 reads past the end: boom
    check_against_rule(value)
Enter fullscreen mode Exit fullscreen mode

Why every safety net missed it

This is the part that should make every backend engineer uncomfortable, because none of these failures were exotic.

First, the validator. CrowdStrike has a Content Validator whose entire job is to check new detection content before release. It had a logic error of its own: it never compared the number of fields in the new content against the number the interpreter could handle. The guard dog was checking IDs at the door while the fence had a hole in it.

Second, the tests. The new template type had passed stress tests back in March. But the tests used wildcard matching on the 21st field, so they never walked the exact path that would crash. The tests proved the happy path worked. The bug lived on the path nobody walked.

Third, the rollout. CrowdStrike deploys its sensor software in stages, rings of machines, watching each ring before widening. Sensible. But Rapid Response Content skipped all of that and went to every machine at once. The riskiest updates, the ones shipped several times a day, had the weakest rollout of anything the company shipped. The file was live for 78 minutes. That was enough.

What I take from it

Validate shapes, not just values. The validator checked the content but never asked the simplest question: does this have more fields than the reader expects? Whenever two systems share a format, the count and the shape are the contract. Check the contract.

Test the exact thing you ship. A wildcard in a test is a hole in the test. If your test data cannot trigger the bug, your test is decoration.

"Just a config change" is a story we tell ourselves. Config is code that runs with the privileges of whatever reads it. This config ran in kernel mode.

Roll out everything in stages, especially the stuff you ship most often. Frequency is not safety. The thing you deploy ten times a day deserves the canary more than the thing you deploy twice a year, because it gets ten times as many chances to be wrong.

CrowdStrike's own fix list reads like this postmortem in reverse: stricter content validation, real test cases for the 21st field, error handling in the interpreter so a bad read can never crash the machine again, and staged rollouts for Rapid Response Content.

The scariest bugs are not clever. This one was a list with 21 items and a reader built for 20. The clever part was everything around it that let such a simple bug reach 8.5 million machines in 78 minutes.

What is the scariest "just a config change" you have ever shipped?

Sources

Top comments (0)