The fiber optic cable provider has informed us the finished the splicing and network connectivity has been restored. We will now close this incident. A full analysis will be posted after we complete a full investigation in the next few days.
Aug 27, 10:00 BST
Softlayer has informed us that there were delays in pulling the new fiber and that the splicing procedure begun only at 04:30 UTC.
Aug 27, 08:34 BST
We have received an update via Softlayer that fiber crews will need to replace 2500 - 3000 ft of cable. The new fiber reel is en route and the crews are expecting to have the new cable spliced by 2014-08-27 2:00 UTC.
Aug 26, 21:14 BST
Service to SDv1 has been restored. All services are now running normally but we are in a state of reduced redundancy due to the fibre cut at Softlayer. We will update this if any further issues arise and when we receive updates from Softlayer.
Aug 26, 20:09 BST
Softlayer are continuing to work on the networking issue affecting connectivity in Washington which is causing reduced redundancy due to a fibre cut. All services are operational except for customers still on our SDv1 platform (users of account URLs ending in .com). We are still investigating the cause of the outage here.
Aug 26, 19:58 BST
The networking issue between our primary and secondary data centers is still ongoing. SDv2 (users of account URLs ending in .io) is fully operational. SDv1 (users of account URLs ending in .com) is currently unavailable - we are investigating the cause.
Aug 26, 19:31 BST
Service has now been restored but the ongoing networking issue affecting internal traffic between our two data centers is still ongoing, which may have further impact on our redundancy. Updates will follow as we have them from Softlayer.
Aug 26, 18:57 BST
As of 26-Aug-2014 15:47 UTC, multiple redundant links failed between the aggregation routers and the backbone routers in our primary data center at Softlayer Washington DC, USA due to a fibre cut. This caused increased latency and packet loss for internal traffic between the primary data center and our secondary data center in San Jose, USA affecting the replication between facilities but not affecting customer service to Server Density.
Softlayer informed us this was being investigated.
At around 17:30 UTC, the situation escalated and our primary facility network became unavailable, affecting customer services. Due to the on-going networking issue and replication lag, we have been unable to fail over to the secondary data center.
All services are effected by this but we are seeing systems recover and so are now starting our post outage checklists to verify all systems. Further updates will follow shortly.
Aug 26, 18:46 BST
Our provider SoftLayer is experiencing a complete unavailability to their Washington data center, our current primary data center. This is causing a complete V1 and V2 service unavailability.
We'll be updating this as we receive more information from SoftLayer.
Aug 26, 18:36 BST