GitLab deleted its production database, then watched five backup layers fail

On May 18, 2017, a GitLab employee attempted to delete a test database on his local development environment. Due to a copy-paste error and a misconfigured SSH session, he executed the `drop database` command against the live production PostgreSQL instance instead. Within seconds, 430 GB of user data — including code repositories, issues, and merge requests — was permanently erased.

The company’s disaster recovery plan relied on five distinct backup layers: daily snapshots, continuous replication, off-site archives, and others. In a catastrophic cascade of failures, every single layer proved corrupted or incomplete. The most recent reliable backup was from four days prior, meaning three days of work by thousands of users vanished without a trace.

In an unprecedented move, CTO Sid Sijbrandij live-streamed the recovery process on Twitch for over six hours. Engineers worked through the night, manually reconstructing data from cached API responses, browser history, and GitHub mirrors where users had forked their projects. While much was recovered, the incident exposed critical fragility in how the company managed its own infrastructure.