We have not seen re-occurrences of this incident from our provider and Server Density systems have been operating as expected since the network connectivity was restored at 07:09 UTC.
We'll share our provider post-mortem as soon as it becomes available.
Posted Jul 20, 2017 - 08:56 BST
Monitoring
We have just received confirmation from our provider: "Backend network connectivity has been restored to affected customer hosts."
We are monitoring the systems closely in case of another occurence.
Posted Jul 20, 2017 - 08:22 BST
Update
Update from our provider "Network connectivity was briefly restored to affected customer hosts however the issue has recurred. Engineers are still working with hardware vendor to mitigate the disruption to customer hosts."
Posted Jul 20, 2017 - 07:59 BST
Update
Update from our provider: "Attempts to restore connectivity to affected customer VLANs have been unsuccessful. Engineers are continuing to investigate and are reaching out to the hardware vendor for assistance."
Posted Jul 20, 2017 - 06:50 BST
Update
Update from our provider: "Engineers have identified that some customer VLANs on bcr01a.wdc04 are down. This is the cause of the disruption for some customer hosts behind bcr01a.wdc04 and Engineers are working to recover service to these hosts as quickly as possible."
Posted Jul 20, 2017 - 05:59 BST
Identified
Update from our provider: "Backend private network connectivity may disrupted for customer in wdc04, pod01. Networking staff are investigating at this time and working to restore connectivity to affected customer hosts."
We are working on restoring services to normal operation.
Posted Jul 20, 2017 - 05:59 BST
Update
One of our providers is performing emergency maintenance which is affecting several of our services:
"Network Engineers have identified a software defect on a redundant backend customer router (BCR), bcr01.wdc04, which provides private connectivity for your server(s). To prevent an unplanned outage resulting in significant downtime, Engineers will be upgrading the firmware during an EMERGENCY maintenance window at the time referenced above."
The period of the maintenance window is: 04:00 UTC - 06:00 UTC
Posted Jul 20, 2017 - 05:53 BST
Investigating
We are currently working on identifying the cause of the stall, during this period you might have issues logging in to the UI and gaps might be visible.