Heroku ran every customer in one AWS region — then EBS broke for 80 hours

On April 21, 2011 at 8:15 UTC, monitoring alerts fired across Heroku: timeouts were hitting its web, caching and routing services. The trigger was upstream — at 12:47 AM PDT, an AWS network change in a single US-East availability zone shifted primary EBS traffic onto a lower-capacity replication network, isolating EBS nodes and setting off a re-mirroring storm that left 13% of the zone's volumes stuck, per AWS's own account.

EC2 itself recovered in about 12 hours, but EBS stayed degraded for more than 80. Heroku — which ran its whole platform in US-East, with its databases on EBS — had nowhere to fail over: there was no second region. Most applications came back within 16 hours; free apps on shared database servers took up to three days.

Heroku's post-mortem refused the obvious excuse: "Heroku takes 100% of the responsibility for the downtime affecting our customers." The three lessons it drew: availability zones in one region do not partition risk the way the team had assumed; block storage is not a cloud-friendly technology; and a platform that promises to abstract infrastructure away owns the consequences when that infrastructure fails.

Most apps returned within 16 hours; free apps on shared databases stayed dark up to 3 days; US-East EBS was degraded for 80+ hours

The whole platform lived in a single AWS region, and its availability zones turned out to share failure modes Heroku had assumed were partitioned.

Stateful services sat on EBS, which stayed broken long after EC2 recovered — a dependency Heroku could neither work around nor replace mid-incident.

The standard playbook — kill unhealthy instances, replace them, wait for AWS — assumed outages last an hour or two. This one lasted days.

Multi-region had been deprioritized for years as a big project with little availability payoff. The April outage repriced it in one weekend.

Availability zones in one cloud region are not redundancy. If every customer runs in the same region on the same block storage, one upstream network mistake becomes everybody's outage.

Heroku said it would treat multi-region as a much higher priority and reduce its dependence on EBS, and began architectural changes meant to let the platform survive IaaS-level failures. AWS's summary of the event became one of the most widely read failure analyses in the industry, and "don't put everything in one region" became standard cloud advice. Heroku nonetheless kept a strongly single-region posture for years afterward, and the concentration it identified kept resurfacing in later incidents.