案例库 · 工程与运营 · 技术决策 · 2024
这条还没译成中文，下面是英文原文。
UniSuper mirrored backups across two Google Cloud regions — one bug deleted both
Google Cloud's misconfiguration deleted UniSuper's $135B pension account in both regions at once; only a backup on a separate provider avoided data loss.
UniSuper · Google Cloud · 2024-05
怎么回事
On May 2, 2024, UniSuper's online accounts, app and phone systems went dark for its 647,000 members. The cause, disclosed in a joint statement with Google Cloud, was an 'inadvertent misconfiguration' during the provisioning of UniSuper's Private Cloud subscription — a rare private-cloud product Google runs on customer-supplier-managed infrastructure. The misconfiguration deleted UniSuper's entire cloud subscription outright, not a single system or dataset within it.
UniSuper had built its redundancy the standard way: the account was duplicated across two separate geographic regions specifically so that a failure in one would not touch the other. The deletion did not respect that boundary. Because the trigger removed the subscription itself rather than data inside it, both regions' copies were deleted together, in the same event — the fault Google's architecture was supposed to isolate.
Google's own systems had no path back. Restoring hundreds of virtual machines, databases and applications took nearly two weeks of joint manual recovery work by UniSuper and Google Cloud engineers, running from May 2 to May 15. What kept the outage from becoming permanent data loss was a decision UniSuper had made independently of Google: it also held backups with a separate cloud provider, outside Google's infrastructure entirely. Those backups minimised data loss and gave engineers a source to rebuild from once the Google-side recovery was under way.
Google Cloud CEO Thomas Kurian called it 'an isolated, one-of-a-kind occurrence' that had never happened to any Google Cloud customer before. The joint statement was blunter about fault: 'This should not have happened. Google Cloud has identified the events that led to this disruption and taken measures to ensure this does not happen again.' Member balances were confirmed up to date once services returned, but two weeks of locked-out retirement accounts, for a fund holding $135 billion in other people's savings, was the visible cost.
为什么会这样
- UniSuper's redundancy plan assumed the two regions failed independently. The actual failure mode — deletion of the parent subscription — sat above that boundary and took both copies down in one event.
- Google Cloud's provisioning tooling for Private Cloud let a misconfiguration delete a live customer's entire account with no confirmation step and no built-in recovery path once it happened.
- Multi-region duplication inside one provider is not independence: it protects against a regional outage, not against a control-plane action that targets the account itself.
- UniSuper survived only because it had paid to maintain a second, out-of-provider backup — a step beyond what its Google Cloud redundancy design alone would have required.
教训
Duplicating across two regions inside one provider protects against a regional outage, not a control-plane mistake that deletes the account itself. Redundancy needs a backup the provider can't reach.
后来呢
Full service was restored on May 15, 2024, with member balances confirmed accurate as of the outage date. Google Cloud stated it had implemented measures to prevent a recurrence of the Private Cloud provisioning fault; neither company disclosed the specific technical fix. The incident became a widely cited case study in cloud architecture for distinguishing multi-region replication from provider-independent backup, and for auditing whether a cloud account's own deletion is a modeled failure mode.
资料来源
- The Register — UniSuper Google Cloud outage: what actually happened
- NotebookCheck — Google Cloud deletes $135B account for UniSuper, then takes two weeks to restore
发现哪里写错了？告诉我们。
类似的案例
这家公司栽倒的地方，别处有人漂亮地解开过。 第二意见 →
Comments · 0
登录 后就能评论。