At 23-Aug-2016 19:26 UTC the SoftLayer Network Operations Center received notifications of connectivity issues to the SoftLayer San Jose Data Centers. Instability was discovered on the SoftLayer Backbone Network path between San Jose and Denver. Network Engineers removed the affected path from production and re-routed network traffic across an alternate path at 23-Aug-2016 20:55.

During the time of this incident, customers located in the West Coast USA may have seen connection issues (latency and dropped packets) when trying to connect to the Server Density UI, API and during postbacks from servers located within the same geographic area. Our monitoring indicated this affected less than 1% of customers.

Trajectory indicated the cause of the instability was with the core backbone switch cbs01.eq01.sjc02. Network Engineers engaged the hardware vendor for a root cause analysis. The hardware vendor discovered a previously unidentified software defect that was triggered by an unrelated configuration change. The conditions were reproduced in a lab environment, and the hardware vendor is developing a fix.

While a fix is being developed Network Engineers and the hardware vendor have determined a process to avoid triggering the software defect. Once Network Engineers have been provided the fix from the hardware vendor and have vetted it against conditions on the SoftLayer Backbone Network, non-disruptive maintenances will be scheduled to deploy it to production.