Facebook's self-healing config checker turned one bad value into a 2.5-hour blackout

In September 2010, Facebook went dark for approximately two and a half hours. The company's engineering team, in a rare public postmortem posted the same day by director of software engineering Robert Johnson, called it the worst outage the site had suffered in over four years. Every visitor, on every page, hit errors until the site came back.

The root cause was an automated system meant to keep configuration values healthy: it compared cached values against a persistent store and replaced anything it judged invalid. That day, a change left a bad value in the persistent store itself. Every client saw the 'invalid' value and tried to repair it, and because the repair meant querying a database cluster, the cluster was hit with hundreds of thousands of queries per second.

The error handling made it worse. When a client's query failed, it read the failure as 'value still invalid' and deleted the matching cache key — generating yet another repair query. Even after the original bad value was fixed, the loop kept feeding itself. The only way out was to stop all traffic to the database cluster, which meant turning the site off entirely, letting the databases recover, and then letting users back in slowly.

Full site down for roughly 2.5 hours — Facebook's own engineers called it the worst outage in more than four years

The verifier assumed the persistent store was always correct; when it held the bad value, every client received the same poisoned repair instruction.

Error handling conflated 'database overloaded' with 'invalid value', so each failure generated more load — an amplifying feedback loop.

Deleting cache keys on error removed the one layer that could have absorbed the retry storm.

There was no circuit breaker between clients and the cluster, so the only remaining lever was a total site shutdown.

Self-healing code that treats every error as 'go fix it' amplifies faults instead of absorbing them. Bound retries per client, dampen feedback loops, and keep a recovery lever short of total shutdown.

Facebook switched off the self-correcting configuration system the same day and said it was redesigning it around patterns that tolerate transient spikes and feedback loops. Service was fully restored that day and the company, then the world's largest social network, published the technical details within hours. The incident is still cited as a textbook case of error handling that turns a small fault into a self-sustaining storm.