Epic's Fortnite buckled under 3.4M concurrent players — six outages in one weekend

On February 3–4, 2018, Fortnite Battle Royale hit a peak of 3.4 million concurrent players — up from a previous peak of 60,000 just months earlier — and the weekend became a rolling outage. Epic's own postmortem, published February 8, 2018, counts six distinct incidents with a mix of partial and total service disruptions.

The deepest cuts were in the MCP profile service, backed by nine MongoDB shards handling 318K reads and 132K writes per second. The matchmaking shard began queueing writes, spiking single operations to 40,000+ milliseconds and blocking every MCP thread; it never self-recovered, so operators performed manual primary failovers several times per hour. A second wound: a packaging change had silently capped the service's thread count below the intended safe value, and the recent 'fix' — reverting it — starved database connection pools only at production peak, forcing a rollback.

On Sunday the Account Service went down from 18:30 to 20:20 UTC: Memcached instability made Nginx block on 100ms timeouts until its worker threads ran out, health checks drowned, and the load balancer pulled every node — signing in was blocked worldwide. That night the XMPP social layer failed during an emergency blue/green failover: the reconnection land rush flattened the Friends Service load balancer, which could not provision replacements because its subnet had run out of free IPs. Players connected but saw every friend as offline — a 'dark room.'

Cloud ceilings made it worse: thousands of c4.8xlarge instances approached the region's instance limit, and /24 subnets offering 251 usable IPs had been exhausted, stretching load balancer recovery. Epic flew MongoDB experts on-site, promised a messaging-stack and MCP re-architecture, and ended the postmortem with a recruiting pitch — a small team of veteran online developers had kept the world's biggest PC/console game online until the numbers outran them.

Six partial and total disruptions across Feb 3–4, 2018: sign-ins blocked, matchmaking frozen, friends shown offline; manual failovers ran hourly.

Matchmaking state lived on one shared shard; when its writer queued, every MCP operation blocked, and the only recovery was a manual primary failover.

A packaging change had capped MCP service threads below the safe default; the correction itself starved database connection pools at production-only peak and had to be rolled back.

All Account Service traffic crossed an Nginx proxy whose workers blocked on 100ms Memcached timeouts; saturated workers failed health checks and the load balancer removed every node.

The failover path for the Friends Service needed free IPs in a /24 subnet — 251 addresses — and the reconnection land rush exhausted them mid-incident.

XMPP ran a full-mesh cluster (101 nodes, ~1,000 sockets each just for cluster links) with a hard node ceiling, so social-layer capacity could not simply be added.

Growth outruns architecture faster than hiring can fix it: at 50× scale, an untested thread setting, a 251-IP subnet and one hot shard each become outages.

The postmortem's fix list reads like a re-architecture program: root-cause the MongoDB write queue with on-site experts, move matchmaking sessions to in-memory storage, route non-verify traffic past the Nginx/Memcached pair, fix subnet capacity, and reduce blast radius by spreading core services across regions and clouds. Epic kept publishing Fortnite service updates as the game kept growing, and the postmortem became a standard citation for how viral growth converts every latent misconfiguration into an outage.