Back to the archive

The encyclopedia · Software & IT · Operational decision · 2023

Optus went dark for 14 hours, cutting off 10 million Australians from everything

A software upgrade triggered a BGP routing collapse that took down Australia's second-largest telco for half a day.

Singtel

What happened

On November 8, 2023, Australia's second-largest telecommunications company suffered a total network failure that lasted 12 to 14 hours, affecting more than 10 million individuals and 400,000 businesses. The outage took down mobile phones, home internet, landlines, EFTPOS terminals, hospital switchboards, and the Triple Zero emergency call service across the country.

The cause was a Border Gateway Protocol (BGP) routing failure triggered by a software upgrade at a Singtel North American exchange. When one router disconnected as a protective measure against routing update overload, approximately 90 edge provider routers followed, setting off a cascading shutdown that the network's automated safeguards could not stop. The outage lasted from 4:05 AM AEDT until late afternoon, with some services recovering only the next day.

The financial damage was severe. Singtel, the Singaporean parent company, lost over AU$2 billion in market value as its share price dropped 4.8% on the Singapore Exchange. Optus CEO Kelly Bayer Rosmarin faced intense criticism for the company's slow and vague communication during the crisis. She resigned a week later, and the company offered affected customers 200 GB of bonus data as compensation.

The Australian government commissioned an independent review led by Dr. Richard Bean, which released 18 recommendations. The Senate also held a public inquiry. The review called for new customer communication rules, a Triple Zero testing regime, and the creation of a dedicated Triple Zero custodian to prevent a recurrence.

Why it happened

  • A single software upgrade cascaded into a national outage because the network's automated protective measures triggered each other in sequence with no human override.
  • Optus had no rapid backup or failover that could bypass the BGP collapse, leaving 10 million customers with no service for half a day.
  • The company's crisis communication was vague and slow, eroding public trust further while the network was still down.
What it costAU$2B+ in market value lost; CEO resignedcostly

The lesson

A single network change can disable an entire country's communications — test routing updates in isolation and keep a manual override that works at scale.

Sources

spotted an error? The club wants to know.

Comments · 0

    Sign in to join the comments.

    More like this

    Somewhere, someone solved the problem this company failed at. 2nd Opinion →