NTT Docomo's migration rollback triggers a 12-hour mobile outage

On October 14 2021, Japan's largest mobile carrier, NTT Docomo, suffered a major network outage that left about 2 million users unable to make calls or use data services. The core disruption lasted around 12 hours, and for some services using older 3G connections the difficulties stretched for up to 29 hours. Voice calls, data, and even emergency calls were affected.

The cause was an engineering failure during planned maintenance. Docomo was migrating its subscriber-registration function to a new system, and when the migration encountered problems the company attempted to roll it back. The rollback put the network's signalling system under unexpected load, causing it to fail and bringing down service for the affected users.

Japan's Ministry of Internal Affairs and Communications (MIC) responded sternly. Having already seen a series of outages across Japanese carriers, the regulator issued a severe warning and administrative guidance to Docomo on November 26 2021, directing the company to strengthen its change-management procedures and prevent a recurrence. Docomo was also expected to report on its remedial measures.

The decision issue sits in change management: a migration of a critical network function was rolled back in a way that overloaded the signalling system, and the failure of that rollback took down core services. The operational decision to proceed with, and then hastily reverse, the change is what turned routine maintenance into a country-wide disruption.