The encyclopedia · Software & IT · Technical decision · 2024
A bad CrowdStrike update blue-screened 8.5M Windows PCs in the biggest IT outage ever
On 19 July 2024 a faulty Falcon sensor config file crashed 8.5 million Windows machines worldwide, grounding flights and idling hospitals, banks and retailers.
CrowdStrike · Microsoft · 2024-07-19
What happened
On July 19, 2024, the cybersecurity firm CrowdStrike pushed a routine configuration update to its Falcon sensor software, which runs deep inside Windows at the kernel level on corporate machines. The update modified a file — Channel File 291, responsible for screening named pipes — that contained a defect causing an out-of-bounds memory read. Instead of failing gracefully, the sensor triggered an invalid page fault that crashed Windows into a blue screen of death and a boot loop.
Because the Falcon sensor runs with high privileges on so many corporate systems, the failure spread almost instantly. Microsoft later estimated that about 8.5 million Windows devices were affected — less than one percent of all Windows machines, but a huge share of the ones that run businesses. Airlines grounded thousands of flights, hospitals fell back to paper, and banks, broadcasters, retailers and government services went down around the world. CrowdStrike had more than 24,000 customers, including roughly 60% of the Fortune 500.
CrowdStrike reverted the faulty file within about an hour, and CEO George Kurtz confirmed it was an accident, not an attack. But machines that had already crashed could not download the fix automatically: IT staff had to reboot each one into safe mode and manually delete the bad file, a process complicated further on machines using BitLocker disk encryption. Full recovery took days. The worldwide financial damage was estimated in the billions of dollars, making it the largest outage in the history of information technology.
Why it happened
- The content update was not adequately validated before it went out to the entire installed base at once, with no staged or canary rollout.
- The sensor runs at the kernel level, so a bug in it crashed the whole operating system instead of being contained.
- CrowdStrike's software gave customers no way to delay or stage content updates, so every machine took the bad file immediately.
- Recovery required manual, per-machine intervention, so the outage lingered long after the fix existed.
The lesson
Software that runs at the kernel can take down the machine it protects. Roll out to a small canary group first, validate content updates like code, and make a bad update fail safe — not fatal.
Aftermath
CrowdStrike apologized, compensated affected customers, and overhauled how it tests and stages content updates, adding canary rings and customer control over rollout timing. The incident reignited debate about the fragility of letting a single kernel-level security product run across the world's critical infrastructure, and about Microsoft's decision to grant such software deep kernel access on Windows. Delta Air Lines, among the hardest hit, reported hundreds of millions of dollars in losses. 'CrowdStrike outage' became shorthand for how one bad update can stop the world.
Sources
spotted an error? The club wants to know.
Comments · 0
Sign in to join the comments.