Memory bugs broke Salesforce's core switch and its failover - 900K subscribers locked out

On January 6, 2009, salesforce.com went dark. Starting at 20:39 UTC (about noon Pacific), customers could not log in or reach their CRM data; The Register reported more than 900,000 subscribers locked out of the applications they run their businesses on. Business Insider put the full outage at 30 to 40 minutes. For a company whose pitch was that businesses could hand their most important data to its cloud, the better part of an hour of total failure landed during the working day.

Salesforce's own explanation, posted afterward: 'a core network device failed due to memory allocation errors. The failure caused it to stop passing data but did not properly trigger a graceful fail over to the redundant system as the memory allocation errors were present on the failover system as well.' In other words, the redundant device failed the same way the primary did, and the platform had to be recovered with manual steps.

The incident also exposed a second-order failure. Salesforce's public 'trust' status dashboard - the page marketing its reliability - was down along with the product, so customers could not tell whether they were the only ones affected. Hundreds turned to Twitter to confirm the outage, an early demonstration that when a SaaS platform fails, its communication layer fails with it unless it is deliberately kept outside the blast radius.

Roughly 40 minutes of total service failure for more than 900,000 subscribers, mid-business-day, with no working failover.

The failover device carried the same memory-allocation defect as the failed primary, so the 'redundant' path never took the traffic.

Failover did not trigger gracefully; restoring full service required manual recovery steps by the operations team.

The public trust dashboard sat inside the same failure domain, leaving customers blind and pushing them to diagnose the outage on Twitter themselves.

Redundancy built from identical parts inherits identical bugs. If the standby unit carries the same defect as the primary, it is a mirror, not a margin - diversify the failover path.

Salesforce posted its explanation on the trust site, said the root cause had been 'addressed by the work-around,' and said its team would keep working with hardware vendors to detail the root cause and identify further patching. The Register and the business press framed the outage as cloud computing's 'dark lining' for a company that sold availability as its core advantage over installed CRM software. Reliability watchers drew a different lesson: everything fails, and what separates operators is whether customers can find out what is happening - communication, not uptime, was the failure.