Rackspace's Dallas data center lost power twice in 8 days - thousands of sites went dark

On June 29, 2009, power failures at Rackspace's Grapevine data center near Dallas - its primary hosting location - took customer servers down for part of the day. Customer sites went dark en masse, including pages run by Justin Timberlake and Michelle Malkin, and the event site Eventbrite. The outage also hit Rackspace's own systems: the company could not answer email or post updates at first, so customers flocked to Twitter to work out whether they were the only ones affected.

Rackspace disclosed the financial hit to the SEC: one-time service credits preliminarily estimated at $2.5-3.5 million. Its blog apology attributed the outage to 'a range of power infrastructure issues'; TechCrunch summarized the sequence as a power failure that eventually tripped up the backup system itself. The company promised customers an action plan: 'We owe you an action plan to prevent this type of thing in the future.'

Eight days later, on July 7, the same facility lost power again - this time from a failed electrical connection that briefly cut network connectivity to some servers. Only after that second failure did Rackspace complete production load tests on its generators, which it said eliminated the problems and returned the data center to normal operating conditions.

$2.5-3.5 million in one-time service credits, disclosed to the SEC, for a power failure that downed customer servers for part of a day.

The Grapevine plant's backup power chain tripped along with the primary feed, so the redundancy that existed on paper failed exactly when it was needed.

Generators and UPS units were serviced after the first outage but were not production load tested until after the second - the fix trailed the failure by eight days.

Rackspace's own support and status channels shared the blast radius: with internal systems impaired, the company could not respond to customers during the first hour.

Backup power that has never carried production load is a hypothesis, not redundancy. Rackspace ran production load tests on its generators only after the second outage in eight days.

Rackspace disclosed the $2.5-3.5 million credit estimate in an SEC filing, apologized on its blog, and promised an action plan. The July 7 repeat forced faster communication - the company blogged and tweeted throughout - and on July 8 it announced it had completed production load tests on the generators and returned the Grapevine facility to normal operating conditions. The plant had already suffered a major outage in November 2007. 'Your lack of redundancy for your own internal operations is highly concerning,' one customer wrote on the company blog.