The encyclopedia · Software & IT · Technical decision · 2021
Facebook locked itself out of the internet for six hours with one bad maintenance command
On 4 October 2021 a routine command disconnected all of Facebook's data centers. Its DNS servers then withdrew their routes, and Facebook vanished for hours.
Meta · 2021-10-04
What happened
On October 4, 2021, at 15:39 UTC, Facebook and everything on it — Instagram, WhatsApp, Messenger, Oculus — went dark worldwide. For the next six to seven hours, billions of people simply could not reach them. The cause was not a cyberattack. During routine maintenance, an engineer ran a command meant to assess the capacity of Facebook's global backbone network. The command went wrong and accidentally disconnected all of Facebook's data centers from that backbone at once.
The design turned a bad command into a catastrophe. Facebook ran its own DNS servers, and those servers were configured to withdraw their BGP routes — the internet's address-book entries — if they lost contact with the data centers. So when the backbone went down, the DNS servers dutifully told the rest of the internet that Facebook's addresses no longer existed. Facebook had, in effect, deleted itself from the internet, and the withdrawal cascaded so thoroughly that even Facebook's own tools couldn't easily reach it to fix it.
The lockout was almost comically total. Facebook's internal systems went down too: employees couldn't send external email, access the corporate directory, or — in a detail that made headlines — open office buildings and conference rooms with their security badges, which depended on the same network. Recovery required a team to physically get into the Santa Clara data center and manually reset the servers. Services came back gradually between about 21:50 and 22:50 UTC. Facebook's shares fell nearly 5% that day, and Mark Zuckerberg's personal wealth dropped by more than $6 billion.
Why it happened
- A single maintenance command, run without adequate safeguards, disconnected all of Facebook's data centers from its backbone at once.
- The DNS servers were designed to withdraw their BGP routes when they lost the data centers — so the failure automatically erased Facebook from the internet's routing tables.
- Recovery tools and access (including physical building badges) depended on the same network that had failed, locking engineers out of the fix.
- There was no out-of-band path to reach the systems and reverse the change quickly.
The lesson
Build your recovery path so it doesn't depend on the thing that just broke. When your DNS, your auth and your door badges all run on the network you just took down, you've locked yourself out.
Aftermath
Facebook's engineering team published a detailed explanation the next day, and the company apologized. The outage — the worst since 2008 — cost an estimated $60 million or more in advertising revenue and knocked billions off its market value, and it briefly drove users to rival platforms. It became a textbook case in network engineering on the dangers of single points of failure, of recovery systems that depend on the systems they're meant to recover, and of how one unguarded command can take down a service used by a third of humanity.
Sources
spotted an error? The club wants to know.
Comments · 0
Sign in to join the comments.