All Systems Operational
Past Incidents
Sep 2, 2014

No incidents reported today.

Sep 1, 2014

No incidents reported.

Aug 31, 2014

No incidents reported.

Aug 30, 2014

No incidents reported.

Aug 29, 2014

No incidents reported.

Aug 28, 2014

No incidents reported.

Aug 27, 2014
Postmortem - Read details
Aug 27, 15:02 BST
Resolved - The fiber optic cable provider has informed us the finished the splicing and network connectivity has been restored. We will now close this incident. A full analysis will be posted after we complete a full investigation in the next few days.
Aug 27, 10:00 BST
Update - Softlayer has informed us that there were delays in pulling the new fiber and that the splicing procedure begun only at 04:30 UTC.
Aug 27, 08:34 BST
Update - We have received an update via Softlayer that fiber crews will need to replace 2500 - 3000 ft of cable. The new fiber reel is en route and the crews are expecting to have the new cable spliced by 2014-08-27 2:00 UTC.
Aug 26, 21:14 BST
Monitoring - Service to SDv1 has been restored. All services are now running normally but we are in a state of reduced redundancy due to the fibre cut at Softlayer. We will update this if any further issues arise and when we receive updates from Softlayer.
Aug 26, 20:09 BST
Update - Softlayer are continuing to work on the networking issue affecting connectivity in Washington which is causing reduced redundancy due to a fibre cut. All services are operational except for customers still on our SDv1 platform (users of account URLs ending in .com). We are still investigating the cause of the outage here.
Aug 26, 19:58 BST
Update - The networking issue between our primary and secondary data centers is still ongoing. SDv2 (users of account URLs ending in .io) is fully operational. SDv1 (users of account URLs ending in .com) is currently unavailable - we are investigating the cause.
Aug 26, 19:31 BST
Update - Service has now been restored but the ongoing networking issue affecting internal traffic between our two data centers is still ongoing, which may have further impact on our redundancy. Updates will follow as we have them from Softlayer.
Aug 26, 18:57 BST
Update - As of 26-Aug-2014 15:47 UTC, multiple redundant links failed between the aggregation routers and the backbone routers in our primary data center at Softlayer Washington DC, USA due to a fibre cut. This caused increased latency and packet loss for internal traffic between the primary data center and our secondary data center in San Jose, USA affecting the replication between facilities but not affecting customer service to Server Density.

Softlayer informed us this was being investigated.

At around 17:30 UTC, the situation escalated and our primary facility network became unavailable, affecting customer services. Due to the on-going networking issue and replication lag, we have been unable to fail over to the secondary data center.

All services are effected by this but we are seeing systems recover and so are now starting our post outage checklists to verify all systems. Further updates will follow shortly.
Aug 26, 18:46 BST
Identified - Our provider SoftLayer is experiencing a complete unavailability to their Washington data center, our current primary data center. This is causing a complete V1 and V2 service unavailability.
We'll be updating this as we receive more information from SoftLayer.
Aug 26, 18:36 BST
Aug 25, 2014

No incidents reported.

Aug 24, 2014

No incidents reported.

Aug 23, 2014

No incidents reported.

Aug 22, 2014

No incidents reported.

Aug 21, 2014
Postmortem - Read details
Aug 27, 15:40 BST
Resolved - This issue is now resolved and we'll be releasing a complete root cause analysis in the next few days.
Aug 21, 19:06 BST
Monitoring - We have forced a failover and restarted the affected component. V2 UI is available again.
We'll closely monitor this for the next few hours while investigating the root cause.
Aug 21, 14:43 BST
Identified - We have identified the issue to be related to our front end component that accepts and handles users connections.
Data collection and alerting does not appear to be affected.
Aug 21, 14:35 BST
Investigating - We are currently experiencing failure to load the V2 UI (.io domains). Investigation is ongoing and we'll update this as we gather more information.
Aug 21, 14:15 BST
Aug 20, 2014

No incidents reported.

Aug 19, 2014

No incidents reported.