A latent bug turned Gmail's routine data-center maintenance into a 3-hour global outage

On February 24, 2009, Gmail went down for users worldwide. Google's incident report for its Apps customers put the disruption at roughly 9AM to 12PM GMT — about three hours — with recovery staggered so individual users lost access for varying periods. No data was lost.

The mechanics were an interaction between two recent decisions. Google had deployed new software to optimize where user data lived. That morning, engineers began routine maintenance in one European data center — an operation designed to be invisible, since traffic shifts automatically elsewhere. But the shift triggered a latent bug in the new locality code: the destination overloaded, and as traffic was automatically re-shifted, overloads cascaded downstream from data center to data center. Engineers took about an hour to get the cascade under control, and longer to rebalance everything.

Google handled the aftermath unusually openly for the era: Gmail Site Reliability Manager Acacio Cruz explained the cascade on the official Gmail blog the same day, and Google published a formal incident report for Apps customers. It also committed to building something that did not yet exist for the service — a public status dashboard — acknowledging that a cloud provider owes customers fast information during an outage, not just silence and a fix.

Gmail inaccessible up to three hours on February 24, 2009, with recovery staggered by data center; no data lost

The failover path — traffic shifted to a second data center — had not been exercised against the new data-locality software, so the latent bug only fired during maintenance.

The overload response amplified the failure: as users were automatically re-shifted, each destination hit the same overload condition, spreading it downstream.

The maintenance design assumed clean handoffs between data centers, but the new optimization code changed the assumptions the failover had been built on.

Communication lagged the incident: Google's own postmortem admitted it needed to acknowledge outages quickly, which led directly to the Apps status dashboard.

Failover paths deserve as much testing as primary ones: new code that rearranges data location turned routine maintenance into a cascade. And a cloud outage calls for a published incident report.

Google fixed the bug and pushed out changes, published the incident report, and launched the Google Apps Status Dashboard weeks later, institutionalizing public outage communication. The 'gfail' episode became an early template for cloud incident reporting — a formal, customer-facing account of cause and remediation — at a time when most consumer web services offered no explanation at all.