Back to the archive

The encyclopedia · Software & IT · Technical decision · 2021

OVHcloud: one fire took two Strasbourg data centres and 3.6M websites offline

A suspected UPS fire destroyed OVHcloud's SBG2 hall and doomed SBG1; 3.6 million websites went dark and recovery ran for months.

OVHcloud · 2021-03-10

What happened

At 00:47 on 10 March 2021 a fire broke out in a room of the SBG2 hall of OVHcloud's data centre campus in Strasbourg, in an industrial area on the River Rhine. Firefighters could not control the blaze: SBG2 was destroyed completely by 04:09, part of SBG1 went with it, and the electricity was cut across the whole site — so SBG3 and SBG4, untouched by flames, went dark too. Nobody was hurt.

The outage was huge because OVHcloud is huge: the largest hosting provider in Europe and the third-largest in the world. Netcraft counted 3.6 million websites across 464,000 distinct domains taken offline, among them 184,000 French sites on 59,600 .fr domains. The victims ranged from the chess server Lichess, the game Rust and the crypto exchange Deribit to the Centre Pompidou, the European Space Agency and Villarreal football club — and customers with no copy of their data outside Strasbourg found the disaster-recovery advice empty.

The response ran for months. OVH promised 10,000 new servers within three to four weeks and replacement infrastructure at its Roubaix and Gravelines sites; more than 100 firefighters, including a pump boat on the Rhine, took over six hours to put the fire out. Cleaning one rack took seven hours. In April the CFO called the restoration 'a real nightmare' and Klaba 'a huge Tetris'. OVH stopped billing affected services and handed the cause to police and insurers: UPS7 had been serviced by its supplier that morning, and UPS7 and UPS8 were both burning when the firefighters arrived.

The scale of the damage settled SBG1's fate: it was damaged so badly it would never be revived. Its racks were cleaned and moved to the surviving SBG3 and SBG4 halls or to Roubaix and Gravelines — a whole data centre written off. One fire in one building had cost OVHcloud two halls, months of 'real nightmare' recovery, and the discovery that a campus is only as safe as its least redundant building.

Why it happened

  • One campus, one switch: four data centres shared one site, and a precautionary power cut took all of them offline — SBG3 and SBG4 burned nothing yet still went dark.
  • The suspected trigger was routine: UPS7 had been serviced by its supplier that morning, and both UPS7 and UPS8 were ablaze when the firefighters arrived — maintenance on a single point of failure.
  • The product assumed the customer had a plan: OVH's first instruction was to activate disaster-recovery plans — for customers with data only in Strasbourg, that last line of defence was empty.
  • Recovery became the second bill: racks cleaned by hand, seven hours each, under a COVID lockdown, with the CFO still calling it 'a real nightmare' a month in.
What it costSBG2 destroyed, SBG1 abandoned; 3.6M websites offlinecostly

The lesson

One fire took an entire campus offline because everything sat in one place. The UPS had been serviced that same morning — and for customers without a backup, the cloud turned out to be a building.

Sources

spotted an error? The club wants to know.

Comments · 0

    Sign in to join the comments.

    More like this

    Somewhere, someone solved the problem this company failed at. 2nd Opinion →