One engineer's config change broke the Twitter API, after layoffs left one SRE on it

Ars Technica reported on 7 March 2023, citing Platformer, that Twitter's outage the day before, which broke links and other functions for about an hour, followed a change in a project to shut down free access to the Twitter API. Twitter said it came from 'an internal change that had some unintended consequences'. Elon Musk blamed code that 'is extremely brittle for no good reason'.

Platformer reported that only one site reliability engineer was staffed on the paid-API project, and that on Monday this engineer made a 'bad configuration change' that 'basically broke the Twitter API', with cascading effects that also took down many internal tools. Public error messages said the user's API plan did not include access to the endpoint, which fed speculation about paid access. An employee said the outage was the sixth major one since 23 January and that people were 'numb' to them.

Ars said the deeper issue was cost-cutting that reduced staff from about 7,500 to under 2,000, with fewer than 550 full-time engineers. A Washington Post article said a risk evaluation team that had vetted product changes before Musk's takeover had been laid off. One employee agreed there was tech debt from the old Twitter, while another said 'this is what happens when you fire 90 percent of the company', with Musk demanding fast changes.

About one hour of broken links and functions; the sixth major outage since 23 January per Platformer.

Only one site reliability engineer was staffed on the paid-API project, according to Platformer.

Twitter's staff had fallen from about 7,500 to under 2,000, with fewer than 550 engineers.

The team that vetted product changes for anticipated problems had been laid off, per the Washington Post.

Musk was demanding fast implementation of major changes, and old code was brittle with tech debt, according to employees.

Cutting the staff who vet and operate risky changes makes ordinary configuration edits dangerous; one person cannot be the safety net.

Musk said the entire code base needed a 'complete rewrite'. Employees said such outages had become frequent enough that they were numb to them. Twitter had not given a fuller account when Ars published.