One missing concurrency limit took Bluesky down for half its users for eight hours

On Saturday 4 April 2026, Bluesky's AppView data plane — the backend serving the social app's reads — started logging errors like 'failed to set post cache item: dial tcp: bind: address already in use'. The data plane leans heavily on memcached to keep load off its main Scylla database, and Jim Calabro, the infrastructure engineer on call, saw the log spikes line up with dips in user-facing traffic. Network monitoring was clean, so the cause stayed hidden through the weekend while service hobbled along.

The root cause was a new internal service deployed that week: under three GetPostRecord requests per second, but 15,000–20,000 post URIs per batch (normal is 1 to 50). Every data-plane RPC handler bounds concurrency with errgroup.SetLimit — this endpoint alone was missing that line. A single call spawned up to 20,000 goroutines, dialing a storm of memcached connections; once the 1,000-socket idle pool overflowed, they piled up in TCP TIME_WAIT and exhausted every ephemeral port. The client ran in one data center, and with no per-client metrics, cache errors of every type masked the culprit.

On Monday 6 April it tipped into a death spiral: with millions of memcached requests failing per second, Bluesky's log-every-error policy attempted millions of blocking write(2) log writes per second. The blocking-syscall flood pushed the Go runtime from about 150 to 1,500 OS threads, starving the GC into long stop-the-world pauses. Aggressively tuned GOGC and GOMEMLIMIT made the data plane OOM-crash periodically — up around 30 minutes, down, repeat. Each restart deepened the hole: TIME_WAIT sockets left the restarted process unable to recreate its memcached pool.

The Monday band-aid was gloriously absurd: a custom dialer that picked a random 127.x.y.z loopback IP for every outgoing connection, expanding the ip+port space beyond the exhausted range. That stopped the spiral, and only on Wednesday did the team find the true root cause and restore the missing SetLimit(50). Calabro's public post-mortem also owned a communication failure: the status page had blamed a third-party network provider, which 'it clearly was not'.

Intermittent downtime for roughly half of all Bluesky users over about eight hours on 6 April 2026 — the platform's worst outage in its infrastructure lead's time there

Bounded concurrency was enforced by convention, not structure: every handler had SetLimit except one, and a single unbounded endpoint was enough to exhaust host-level resources.

The traffic contract had no shape limits: nothing rejected a 20,000-item batch, so one quiet internal client — 3 requests a second — outranked millions of normal ones as a load source.

Failure handling amplified the failure: logging every memcached error at millions of errors per second converted a cache problem into thread explosion, GC pauses, and OOM crashes.

Observability assumed small requests and had no per-client metrics, so errors from every cache type hid the one caller and one endpoint responsible for days.

Restart-as-recovery was broken: TIME_WAIT sockets from the dead process blocked the new process from rebuilding its memcached pools, turning OOM restarts into more port exhaustion.

One unbounded goroutine loop outweighed every other safeguard. Concurrency limits must be structural, not a per-endpoint habit: the single endpoint missing one became the whole incident.

Service was stabilized on Monday with the random-loopback dialer, which was removed once the real fix — restoring the concurrency limit — landed on Wednesday 8 April. The total exposure was about eight hours of intermittent downtime for roughly half of users. The post-mortem, published 10 April 2026, promised per-client observability, detection of clients sending small numbers of oversized requests, and a shift from high-volume logging toward Prometheus metrics and OpenTelemetry tracing; it drew a detailed independent analysis by reliability researcher Lorin Hochstein two days later.