Vipshop's cooling failure cost ¥100M and a VP his job

On March 29, 2023, Vipshop's e-commerce platform suffered a catastrophic outage when a cooling system failure at its Nansha IDC (Internet Data Centre) caused server room temperatures to rise rapidly, triggering a cascade of equipment shutdowns. The outage lasted over 12 hours, affecting more than 8 million customers and causing the company over ¥100 million in direct losses. Vipshop classified the incident as a P0 (most severe) level failure.

The company's response was swift and public: the head of the infrastructure platform department was fired, and the direct managers of the affected teams were held accountable. The incident also impacted Tencent, which hosted services in the same data centre and internally classified the event as a 'Level 1 accident,' resulting in criticism, demotions, and dismissals of several executives. The failure exposed the risks of single-data-centre architecture and inadequate disaster recovery planning.