A routine upgrade at Rogers took down Canada's networks for 19 hours — 911 went with it

On July 8, 2022, at about 4:30 a.m. EDT, Rogers Communications' network began failing across Canada. By 4:58 a.m. the core network had crashed: more than 12 million customers lost wireless and wireline service — mobile, home internet, corporate and institutional lines — and about a quarter of Canada's observable internet connectivity disappeared. Most services were restored within 15 to 19 hours, but full restoration took until 7:00 a.m. on July 9, and some customers stayed unconnected for days.

The independent assessment the CRTC commissioned, released in July 2024, traced the failure to a routine change. During phase six of a seven-phase upgrade of its IP core, Rogers staff removed the Access Control List filter from the distribution routers' configuration. With the filter gone, one distribution router released more than 900,000 route advertisements into the core — against roughly 10,000 normally — and the core crashed within minutes. A risk-assessment algorithm downgraded the phase to 'Low' risk after five clean phases, which the CRTC said contravenes industry norms.

The redundancy that should have saved Rogers was not independent. Wireless and wireline ran on one common IP core, so when the core failed, both went down together. Worse, Rogers' management network ran on that same core: staff at the network operations centre lost remote access, had no backup connectivity from an alternative provider, and could not read the failed routers' error logs — it took about 14 hours to pinpoint the cause.

The outage rippled through the economy: Interac debit and e-Transfer failed nationwide, banks reported ATM and online problems, Tim Hortons went cash-only, and police in Toronto and Ottawa said some callers could not reach 911. Rogers estimated it would refund about C$150 million — five days of credit for its roughly 13 million customers — and shares fell 4.61% on the first trading day after. The CRTC opened an examination, and the outage injected new regulatory risk into Rogers' pending Shaw acquisition.