The encyclopedia · Engineering & Operations · Technical decision · 2017
A contractor switched off a power supply — British Airways lost 3 days and $100M
A contractor accidentally cut power to BA's data centre. The uncontrolled reconnection surged and destroyed servers, grounding the airline for three days.
British Airways · 2017-05-27
What happened
On 27 May 2017, a bank holiday weekend, a maintenance contractor at British Airways' Boadicea House data centre near Heathrow accidentally switched off the uninterruptible power supply feeding the site's servers.
The power was then reconnected, but not gradually as the UPS design assumed — it came back in an uncontrolled surge that physically damaged servers and network equipment across the site, an outcome BA's own backup power design had not been built to survive.
The airline cancelled around 800 flights over three days, stranding roughly 75,000 passengers at Heathrow and Gatwick. IAG chief executive Willie Walsh called it human error and said a mistake in disconnecting power was plausible, but reconnecting it in that manner was harder to explain. Analysts put the total cost near $100 million.
Why it happened
- A single contractor's action at one site was able to take down BA's operations because the backup power path had no control against an uncontrolled reconnection surge.
- The UPS system existed to prevent exactly this kind of outage, and the failure mode that destroyed the servers was the reconnection sequence, not a lack of backup power itself.
- Three days of cancellations from one data centre incident show the operation had no tested failover to run from while the primary site was rebuilt.
The lesson
A backup power system that survives losing power but not regaining it protects against the wrong failure. The reconnection sequence needed the same design attention as the disconnection.
Aftermath
BA commissioned an independent investigation into the power failure. IAG absorbed roughly $100 million in compensation, rebooking and lost revenue, and the airline faced sustained criticism over its IT resilience.
Sources
spotted an error? The club wants to know.
More like this
Ericsson's single-sourcing bet: one plant fire killed the mobile phone business
TSMC's Arizona mega-fab costs US$165 billion and still runs 30% dearer than Taiwan
Spain and Portugal went dark for a day when the Iberian grid lost voltage control
Somewhere, someone solved the problem this company failed at. 2nd Opinion →
Comments · 0
Sign in to join the comments.