Alibaba Cloud's access-key bug broke its console worldwide — then the spinoff died

On the Sunday after its Double 11 shopping festival, 12 November 2023, Alibaba Cloud's product consoles and control APIs went dark across every region. The company's own health panel showed all 92 cloud products in an abnormal state, and consumer apps including Taobao, Xianyu, DingTalk and AliDrive broke for users, alongside third-party services running on Alibaba Cloud.

The official fault report traced the outage to the Access Key (AK) service, the component that authenticates control-plane requests. Reading its whitelist data the service hit a read anomaly, and the code meant to handle that exception had a logic defect: instead of failing safe it generated an incomplete whitelist, so valid requests not on the list were rejected. Object storage, table store, log service and message service depended on the AK service and partially failed; already-running compute and database workloads were unaffected.

Engineers fixed the whitelist version and restarted the AK service in batches. Consoles came back region by region from 18:35, most regions recovered by 19:20 — 1 hour 41 minutes after the fault began at 17:39 — and the last affected products were restored by 21:11.

It was Alibaba Cloud's second flagship failure in a year, after the December 2022 Hong Kong region outage. Four days later Alibaba cancelled the planned full spin-off and IPO of its cloud unit, officially citing US chip export controls. On 27 November the platform suffered a second outage in a month, hitting database products in eight regions.

Console and control API down 1h41m; full recovery took 3.5 hours; SLA compensation vouchers estimated above ¥100M — roughly a month of the unit's profit.

Authentication is a shared dependency: the access-key service sits under every product, so a fault there fails them all simultaneously.

The exception handler's logic defect turned a read anomaly into an incomplete whitelist instead of falling back to a known-good copy.

Recovery was manual — repairing the whitelist and restarting the AK service in batches — because there was no automatic rollback path.

It followed a year of churn: two CEO changes and cost cuts at the cloud unit, which Chinese industry analysts linked to weaker operational discipline.

A cloud is only as strong as its shared identity plane: one defective exception handler in the access-key service failed every product at once. Shared foundations need canaries and fallbacks.

Alibaba Cloud paid customers compensation vouchers under its SLA, which promises 30% of a monthly fee once downtime exceeds 43 minutes; individual developers who accrued less were topped up to ¥50. Chinese industry estimates put the total above ¥100 million, about one month of the unit's profit. The spin-off decision followed on 16 November, with CEO Eddie Wu saying a full spin-off 'may not achieve the intended effect'. The 27 November outage hit database consoles and OpenAPI in eight regions including Beijing, Shanghai, Hong Kong and Virginia for nearly two hours.