One fiber cut killed both of Wikipedia's 'redundant' data-center links for 63 minutes

On 6 August 2012 at 6:15am PDT, Wikimedia's operations team was alerted that network connectivity between its two data centers — Ashburn, Virginia, which served most traffic, and Tampa, Florida, which hosted backend services like the database — had been severed. A third-party construction crew working in the Tampa area had cut the fiber. With Ashburn unable to reach its own backend, every Wikimedia site went dark from 13:15 to 14:18 UTC; the mobile site took until 15:35 UTC to come back. Traffic was rerouted to Tampa while repairs went on.

The uncomfortable part was the design review that followed. Wikimedia had contracted for two independent, segregated DWDM systems out of the carrier's Tampa POP, each routed over diverse fibers through separate entrances — redundancy on paper. The post-incident investigation found that both 'diverse' waves traversed one folded fiber segment, and that the metro access segment of wave #1 had been mis-routed onto the same side as the long-haul segment of wave #2. One cut, therefore, removed both connections between the data centers at once.

The outage exposed a second assumption: Ashburn serving 'most of the traffic' still depended on Tampa for core backend services, so losing the link meant losing the site, not just a path. Wikimedia's own incident documentation put it plainly — the fix was to make the carrier audit the waves so 'such single point of failure is no longer in their system', and to finish replicating backend services into Ashburn for true two-site redundancy.

All Wikimedia sites worldwide down for roughly 63 minutes (13:15–14:18 UTC), with the mobile site dark for another 77 minutes; traffic rerouted to Tampa during repairs

The carrier's two 'diversely routed' DWDM waves converged inside a single folded fiber segment in the Tampa metro — a hidden single point of failure inside a redundant design.

The metro segment of wave #1 was incorrectly routed on the same side as the long-haul segment of wave #2, so the construction cut found both at once.

The services ran unprotected across the carrier network, with no automatic protection switching, so a physical cut meant hard downtime rather than a sub-second blip.

Ashburn depended on Tampa for backend services such as the database, which converted one link failure into a full sitewide outage.

Route diversity was taken from the contract and the diagram rather than verified against where the cable actually ran.

Redundancy is a property of the network as built, not of the invoice. Audit the actual fiber routes you are paying for — the carrier's design diagram and the glass in the ground can quietly disagree.

Wikimedia asked the carrier to audit its waves to ensure the single point of failure was eliminated, and accelerated replicating and migrating the remaining backend services to Ashburn (Eqiad) to create full service redundancy across the two data centers. Director of Technical Operations CT Woo said traffic would only revert to Ashburn once the team was comfortable with the provider's fix. The incident doc stayed public on Wikitech as permanent incident documentation.