A 2011 storage bug emptied Gmail inboxes; only offline tape backups brought them back

On 27 February 2011, Gmail users began logging in to find their inboxes empty. The cause was a routine storage software update that carried an unexpected bug. Google stopped the deployment as soon as the problem surfaced and reverted to the old version, but the damage was done: mail had been deleted from accounts representing, at first estimate, 0.08% of Gmail's user base — roughly 150,000 accounts, later revised down to 0.02%.

The uncomfortable part of the postmortem was why the copies hadn't helped. Gmail keeps multiple copies of every mailbox in multiple data centers. But replicas are not independent when they run the same software: 'in some rare instances software bugs can affect several copies of the data,' Google's VP Engineering Ben Treynor wrote. The only clean copies were on tape — offline, and therefore immune to the bug. Tape is slow, which is why restoration took around 30 hours instead of milliseconds; the remaining 0.012% of affected users were restored from tapes by 1 March.

Google stressed that 'email was never lost,' and by the standards of data-loss incidents the story ended well. But the episode punctured the intuition that cloud replication means safety, and complaints about missing mail and login problems continued into a fourth day. It remains the standard citation for correlated-replica failure: redundancy guards against machines dying, not against the same bug running everywhere.

Initial estimate 0.08% of users (about 150,000 accounts, later revised to 0.02%) saw empty inboxes; tape restoration took about 30 hours.

The storage update introduced a bug that could delete data, and deployment reached live systems before the defect was caught.

Gmail's multiple copies all ran the same storage software, so one bug could touch several replicas across data centers.

The only copy the bug could not reach was offline tape — a recovery path Google kept despite replication, which is what saved the mailboxes.

Restoration from tape was inherently slow, stretching a single bad deployment into days of user-visible damage.

Replicas protect against hardware failure, not software. A bug can corrupt several copies at once, so keep one recovery path outside your software's blast radius — for Gmail, offline tape.

Google completed restoration from tape and published an unusually frank account of the failure on the Official Gmail Blog. The incident became a standing argument in reliability engineering for keeping at least one recovery medium outside the failure domain of production software — tape entered the cloud era's vocabulary because of it. Gmail's replication model itself was unchanged; what changed was the caution around correlated software faults in multi-copy systems.