BB&T's 2018 storage failure cut off millions for 15 hours; it blamed Hitachi Vantara

BB&T said an outage that began at about 4 p.m. on February 22, 2018 kept millions of customers from online banking, mobile, ATM and wire transfer services for 15 hours over several days. Some services were not restored until February 26, drawing angry posts on Twitter. The most affected areas included Raleigh, Charlotte, Atlanta, Washington, D.C. and Philadelphia.

BB&T extended branch hours on February 23, 24 and 26, added staff in customer care centers, branches and social media teams, and posted a 97-second video in which CEO Kelly King apologized and explained the steps ahead. In April 2018 CFO Daryl Bible said the outage cost about $15 million in lower deposit service charges and about $5 million in higher operating expenses. King said BB&T had spent $300 million to build redundant data systems to prevent a repeat.

In late 2019 BB&T sued Hitachi Vantara in federal court in North Carolina, calling the outage catastrophic. It alleged breach of contract, unfair and deceptive trade practices and gross professional negligence in the vendor's management of hardware at the Zebulon data center. BB&T said it hired Hitachi in August 2014 for storage disk array equipment and services for its mainframe systems, and that its own employees were not allowed to maintain the array.

A component of the storage disk array at BB&T's Zebulon data center failed, according to BB&T's description of the vendor-managed equipment.

BB&T alleges the vendor that manufactured, installed and maintained the component was responsible for the failure.

BB&T's CEO said the bank spent $300 million afterward building redundant data systems to prevent a repeat.

Hitachi Vantara said it believed it had provided high-quality service at all times, so responsibility remained disputed.

Redundancy has to be built before the failure, not after. A single storage array that a vendor alone is allowed to maintain can take down a whole bank's channels.

In the December 2019 suit BB&T sought $75,000 in damages plus compensatory and punitive damages with interest. Hitachi Vantara said it was reviewing the allegations. The suit came as BB&T was completing its merger with SunTrust to form Truist.