The encyclopedia · Engineering & Operations · Operational decision · 2024
Alibaba Cloud lost a whole zone to a lithium battery in its landlord's data centre
A battery explosion at the Digital Realty SIN11 building in Singapore knocked out Alibaba Cloud's entire availability zone C, disrupting ByteDance and Lazada.
Alibaba Cloud · Digital Realty · 2024-09-10
What happened
On 10 September 2024 a fire broke out at the Digital Realty SIN11 data centre at 3 Loyang Way in Singapore, triggered by a lithium-ion battery that exploded on the third floor of the four-storey building. About twenty people were evacuated and no injuries were reported.
SIN11's most prominent tenant was Alibaba Cloud, which ran its Singapore availability zone C there. The company detected a network-access anomaly in the zone at 10:20 a.m. Beijing time and began moving traffic away, switching most network and security products to other zones by 10:55 a.m.
Recovery was limited because engineers could not enter the building. By midday the high-availability versions of RDS, OSS, OTS and MaxCompute had recovered, but at 8:04 p.m. the fire alarm had not fully cleared and Alibaba warned of a risk of a complete AZ C network outage.
Digital Realty shut off part of the site's electrical systems at 1:45 a.m. on 11 September, and on 12 September Singapore's Building and Construction Authority found minor structural damage and issued a dangerous-building order and a closure order for parts of the facility. Tenants including ByteDance and Lazada were significantly disrupted.
Why it happened
- Alibaba's Singapore region used availability zones to survive the failure of a single building, but a fire is not routine: the zone could not be re-entered, so engineers could not reach the hardware.
- The battery that ignited was the landlord's, outside Alibaba's control, yet the tenant absorbed the outage — a cloud provider's availability promise can be voided by the building under it.
- The gap between detecting the anomaly and fully draining the zone reflects the fact that some services could not be failed over without access to the affected hardware.
The lesson
A cloud region is only as available as the building it sits in. Design for the failure of an entire data centre, not just a server, and assume a fire can keep your engineers out.
Aftermath
Alibaba moved affected services to other zones and worked through a multi-day recovery while Digital Realty coordinated with Singapore regulators. The incident underscored how a single facility can take down a major cloud tenant's entire availability zone.
Sources
- Frontier Enterprise — Alibaba Cloud hit by Digital Realty fire in Singapore
- Channel News Asia — SCDF operation at fire at Loyang Digital Realty data centre
- The Edge Singapore — Fire at Digital Realty's Loyang Way data centre impacts tenant Alibaba Cloud
spotted an error? The club wants to know.
More like this
TSMC's Arizona mega-fab costs US$165 billion and still runs 30% dearer than Taiwan
A chiller failure in an Aurora data centre froze global futures trading for 10 hours
No software fix for a battery that burns — 320,000 Jeeps parked outside
Somewhere, someone solved the problem this company failed at. 2nd Opinion →

Comments · 0
Sign in to join the comments.