A vendor software update took Norfolk Southern's network down for weeks

Supply Chain Dive reported on 1 September 2023 that Norfolk Southern expected to need weeks to recover its backlog of trains after a service outage halted operations earlier that week. The railroad confirmed the outage was not a cybersecurity incident but an error in vendor software.

'During a routine maintenance procedure being performed by our vendor, a defect in their software created an error which caused both our primary and recovery data storage systems to become unresponsive,' the railroad said in a statement, adding that this in turn affected its core operational systems.

Systems were restored and service resumed on Monday evening, with Norfolk Southern working through the backlog and expecting to fully recoup its network in the coming weeks. The railroad said it was working closely with the vendor on steps to prevent recurrence and to identify opportunities to make Norfolk Southern more resilient.

Core operational systems stopped; service was halted and resumed only Monday evening, with the railroad saying full recovery of the backlog could take weeks.

The primary and the recovery storage shared the same failure mode.

Routine maintenance had write access deep enough to corrupt both.

A railroad's dispatch depends on core operational systems that cannot idle.

Backlog on a rail network compounds: trains recover slower than systems.

A recovery system that can be killed by the same defect as the primary is not redundancy — maintenance access needs the same isolation as production.

The railroad said it was working with the vendor to strengthen systems and identify resilience improvements; no financial impact was quantified in the statement.