The encyclopedia · Software & IT · Technical decision · 2017
A mistyped command at AWS took down S3 — and a big chunk of the internet — for four hours
On 28 Feb 2017 an AWS engineer debugging S3 mistyped a command and removed too many servers. S3 — and much of the internet built on it — went down for ~4 hours.
Amazon · Amazon Web Services · 2017-02-28
What happened
On the morning of February 28, 2017, an engineer at Amazon Web Services was debugging a problem that was making the billing system of Amazon S3 — the company's hugely popular cloud storage service — respond slowly. Following a standard playbook, the engineer ran a command intended to take a small number of S3 servers out of service so they could be examined. But a single incorrect input to that command removed far more servers than intended, including servers that ran two of S3's core subsystems: the index that tracks where all the data lives, and the system that allocates storage.
With the index subsystem down, S3 in the US-EAST-1 region (Northern Virginia) couldn't serve requests, and the error rippled outward. An enormous amount of the internet is built on S3 — not just websites storing images and files, but other AWS services too, from computing dashboards to storage volumes to serverless functions. As S3 faltered, those services faltered with it. For roughly four hours, a large swath of popular sites and apps — from Slack and Trello to countless others — were broken or degraded, and the outage trended worldwide.
AWS even struggled to tell people what was happening: its own service-health dashboard depended on S3, so it couldn't post a clear status update while the service was down. Engineers restarted the subsystems and S3 came back online later that morning. Amazon published a candid post-mortem and apologized, and said it would add safeguards so a command can't remove capacity below a minimum, partition the S3 subsystems so one error can't take them all down, and move the status dashboard off the service it reports on.
Why it happened
- A single incorrect input to a routine maintenance command removed more servers than intended, including two of S3's core subsystems.
- There was no guardrail preventing the command from taking critical subsystem capacity below a safe minimum.
- S3's subsystems weren't sufficiently partitioned, so one error cascaded across the whole service in the region.
- So much of the internet — and of AWS itself — depends on S3 that the failure rippled far beyond storage, and even AWS's own status page went down with it.
The lesson
The more of the internet depends on your service, the more dangerous an unguarded command becomes. Guard destructive operations, partition subsystems, and never let one input take down the whole.
Aftermath
The February 2017 S3 outage became one of the most famous cloud failures in history and a textbook case in how a single unguarded command can ripple across an interconnected system. Amazon's transparent post-mortem — explaining the typo, the cascade, and the fixes — is itself often cited as a model of how to communicate after an incident. The episode hardened the industry's focus on guardrails for destructive operations and partitioning critical subsystems, and on the uncomfortable truth that when one service underlies much of the internet, one bad morning there is a bad morning for everyone.
Sources
spotted an error? The club wants to know.
Comments · 0
Sign in to join the comments.