We have confirmed the previous fix has completely resolved this incident.
Posted Sep 16, 2016 - 21:34 BST
Monitoring
We have identified one server which was causing errors into payload processing. That has been fixed, we have started to see regular monitoring data from the affected cluster and we have turned no data alerts back on.
All other services have been mostly unaffected for the duration of this incident except for a small set of devices that will not show data for short periods of time between 14:10 and 17:55.
Posted Sep 16, 2016 - 20:35 BST
Update
We continue to work on the issue. No data alerts remain disabled.
Posted Sep 16, 2016 - 19:26 BST
Update
We are still working on the issue. No data alerts remain disabled.
Posted Sep 16, 2016 - 18:23 BST
Update
We are still working on the issue as the root cause seems to be network degradation at our provider. No data alerts remain disabled.
Posted Sep 16, 2016 - 17:47 BST
Update
We are still working on the issue. No data alerts remain disabled.
Posted Sep 16, 2016 - 17:14 BST
Identified
We have identified an issue on one of our clusters which is causing degraded payload processing performance. We are currently working to restore it.
Posted Sep 16, 2016 - 16:38 BST
Update
We are still investigating this issue. Alerting and UI is not affected. The impacted devices will not show data for the duration of the alert.
Posted Sep 16, 2016 - 16:15 BST
Investigating
We have just disabled no data alerts as we're seeing a high number of those going out.