We have not seen re-occurrences of this incident from our provider and Server Density systems have been operating as expected since the network connectivity was restored at 07:09 UTC.
We'll share our provider post-mortem as soon as it becomes available.
Jul 20, 08:56 BST
We have just received confirmation from our provider: "Backend network connectivity has been restored to affected customer hosts."
We are monitoring the systems closely in case of another occurence.
Jul 20, 08:22 BST
Update from our provider "Network connectivity was briefly restored to affected customer hosts however the issue has recurred. Engineers are still working with hardware vendor to mitigate the disruption to customer hosts."
Jul 20, 07:59 BST
Update from our provider: "Attempts to restore connectivity to affected customer VLANs have been unsuccessful. Engineers are continuing to investigate and are reaching out to the hardware vendor for assistance."
Jul 20, 06:50 BST
Update from our provider: "Engineers have identified that some customer VLANs on bcr01a.wdc04 are down. This is the cause of the disruption for some customer hosts behind bcr01a.wdc04 and Engineers are working to recover service to these hosts as quickly as possible."
Jul 20, 05:59 BST
Update from our provider: "Backend private network connectivity may disrupted for customer in wdc04, pod01. Networking staff are investigating at this time and working to restore connectivity to affected customer hosts."
We are working on restoring services to normal operation.
Jul 20, 05:59 BST
One of our providers is performing emergency maintenance which is affecting several of our services:
"Network Engineers have identified a software defect on a redundant backend customer router (BCR), bcr01.wdc04, which provides private connectivity for your server(s). To prevent an unplanned outage resulting in significant downtime, Engineers will be upgrading the firmware during an EMERGENCY maintenance window at the time referenced above."
The period of the maintenance window is: 04:00 UTC - 06:00 UTC
Jul 20, 05:53 BST
We are currently working on identifying the cause of the stall, during this period you might have issues logging in to the UI and gaps might be visible.
Jul 20, 05:45 BST