Back to the archive

The encyclopedia · Engineering & Operations · Operational decision · 2024

Alibaba Cloud lost a whole zone to a lithium battery in its landlord's data centre

A battery explosion at the Digital Realty SIN11 building in Singapore knocked out Alibaba Cloud's entire availability zone C, disrupting ByteDance and Lazada.

Alibaba Cloud · Digital Realty · 2024-09-10

What happened

On 10 September 2024 a fire broke out at the Digital Realty SIN11 data centre at 3 Loyang Way in Singapore, triggered by a lithium-ion battery that exploded on the third floor of the four-storey building. About twenty people were evacuated and no injuries were reported.

SIN11's most prominent tenant was Alibaba Cloud, which ran its Singapore availability zone C there. The company detected a network-access anomaly in the zone at 10:20 a.m. Beijing time and began moving traffic away, switching most network and security products to other zones by 10:55 a.m.

Recovery was limited because engineers could not enter the building. By midday the high-availability versions of RDS, OSS, OTS and MaxCompute had recovered, but at 8:04 p.m. the fire alarm had not fully cleared and Alibaba warned of a risk of a complete AZ C network outage.

Digital Realty shut off part of the site's electrical systems at 1:45 a.m. on 11 September, and on 12 September Singapore's Building and Construction Authority found minor structural damage and issued a dangerous-building order and a closure order for parts of the facility. Tenants including ByteDance and Lazada were significantly disrupted.

Why it happened

  • Alibaba's Singapore region used availability zones to survive the failure of a single building, but a fire is not routine: the zone could not be re-entered, so engineers could not reach the hardware.
  • The battery that ignited was the landlord's, outside Alibaba's control, yet the tenant absorbed the outage — a cloud provider's availability promise can be voided by the building under it.
  • The gap between detecting the anomaly and fully draining the zone reflects the fact that some services could not be failed over without access to the affected hardware.
What it costavailability zone C offline ~2 days; ByteDance, Lazada hitcostly

The lesson

A cloud region is only as available as the building it sits in. Design for the failure of an entire data centre, not just a server, and assume a fire can keep your engineers out.

Aftermath

Alibaba moved affected services to other zones and worked through a multi-day recovery while Digital Realty coordinated with Singapore regulators. The incident underscored how a single facility can take down a major cloud tenant's entire availability zone.

Sources

spotted an error? The club wants to know.

Comments · 0

    Sign in to join the comments.

    More like this

    Somewhere, someone solved the problem this company failed at. 2nd Opinion →