Cooling failure forced Wikipedia's failover — and the failover itself broke DNS worldwide

On 24 March 2010, cooling at Wikimedia's European data center failed and many servers switched themselves off to protect their hardware. Because that site served European traffic for Wikipedia and the other projects, the operations team ran its standard quick failover: change DNS entries so all user traffic moved to the Florida cluster. Minutes later the team found the failover mechanism itself was broken, and DNS resolution for every Wikimedia site had stopped working — not just in Europe, but globally.

Recovery then hit a second, quieter failure mode. The team expected resolvers to drop the bad answers within an hour, the negative cache TTL the zone advertised, but many ISPs' resolvers ignored that hint and cached the failed lookups longer. The workaround was blunt: rename the affected DNS record so resolvers had to chase a fresh name. Meanwhile the secure.wikimedia.org SSL gateway, disabled during the emergency, collapsed under returning traffic and stayed out until 21:32 UTC.

For several hours the world's reference encyclopedia was simply unresolvable, with no outage stream to watch — Mashable noted Twitter was the only place to track it. The Wikimedia Signpost logged the site 'globally offline for several hours,' and the team posted a same-day explanation on the tech blog, signed by infrastructure engineer Mark Bergsma.

Every Wikimedia site stopped resolving worldwide for several hours, with the outage covered by CNN, the Telegraph and Mashable as readers lost access.

The failover procedure was labeled 'standard quick,' but its DNS step was broken when actually run — the recovery path had evidently never been exercised against a real switch.

Recovery estimates assumed ISP resolvers would honor the advertised one-hour negative cache TTL; many didn't, stretching a bounded incident into an open-ended one.

The SSL gateway had no headroom: when traffic rushed back, secure.wikimedia.org overloaded and dropped, compounding the outage.

European serving was concentrated in a single facility, so one cooling failure mandated a global traffic switch in the first place.

A failover path that is never exercised is a hope, not a mechanism. Rehearse recovery procedures as live drills, and never assume downstream caches expire on schedule.

Access came back after the DNS record was renamed and fresh lookups propagated; the SSL gateway returned by 21:32 UTC the same day. The same-day public postmortem on the Wikimedia tech blog set the tone for the foundation's later habit of publishing incident retrospectives. The redundancy trap itself persisted: in August 2012 a single fiber cut severed both 'redundant' links between the Ashburn and Tampa data centers and took Wikipedia down for another 63 minutes, and only then did the foundation consolidate its serving back end at Ashburn.