Vipshop's cooling failure cost ¥100M and a VP his job

On March 29, 2023, Vipshop's e-commerce platform suffered a catastrophic outage when a cooling system failure at its Nansha IDC (Internet Data Centre) caused server room temperatures to rise rapidly, triggering a cascade of equipment shutdowns. The outage lasted over 12 hours, affecting more than 8 million customers and causing the company over ¥100 million in direct losses. Vipshop classified the incident as a P0 (most severe) level failure.

The company's response was swift and public: the head of the infrastructure platform department was fired, and the direct managers of the affected teams were held accountable. The incident also impacted Tencent, which hosted services in the same data centre and internally classified the event as a 'Level 1 accident,' resulting in criticism, demotions, and dismissals of several executives. The failure exposed the risks of single-data-centre architecture and inadequate disaster recovery planning.

Vipshop ran on a single data centre with no failover, so one cooling breakdown cost ¥100M+ and the VP his job.

Vipshop ran its entire platform on a single data centre with no real-time failover — a cooling failure in one room took down the whole company

The outage lasted 12 hours, suggesting that even basic disaster recovery procedures were not in place or could not be activated quickly enough

A single data centre is a single point of failure, and a cooling breakdown is not a freak event — it is a predictable risk every platform should survive