Update - Our provider has completed emergency maintenance on a router identified as showing backplane congestion through this incident and affecting our network traffic.
We will monitor this throughout tomorrow to confirm the fix.
Sep 21, 22:17 BST
Monitoring - Payload intake and processing has just normalized. We still don't have a resolve statement from our provider so we will continue to monitor this issue for another 24h.
Sep 21, 21:19 BST
Identified - Our provider has just notified us they have identified the issue and are currently working on correction.
Sep 21, 19:30 BST
Update - Our provider continues to work on the issue. We have deployed a code change to minimize impact on device payloads processing which will minimize gaps on graphs for the small number of devices impacted.
Sep 21, 18:20 BST
Update - We're continuing to work with our provider to resolve this. We'll share more significant updates as and when we have them.
Sep 21, 17:09 BST
Update - We're continuing to work with our provider to resolve this.
Sep 21, 15:23 BST
Investigating - We're observing further problems with processing agent postbacks and we're working with our provider on the network degradation which is causing this incident.
Sep 21, 14:40 BST
Monitoring - As during the previous occurrences of this issue, payload intake has normalized and nodata alerts have enabled.
Our provider has not found the cause of this problem yet, they will continue to work on this. We'll continue to monitor the postback intake and processing closely.
Sep 20, 21:05 BST
Update - Our provider networking team is still looking into the cause of the degradation we're observing.
Sep 20, 20:02 BST
Update - We are continuing to work with our provider to track down the network performance degradation we are observing.
Sep 20, 18:00 BST
Investigating - We are observing a number of received device payloads lower than normal again and nodata alerts are disabled at the moment. The affected devices will show gaps on their graphs.
Our monitoring has also picked up degradation on the internal networking and we are currently working with our provider to find the cause and fix it.
Sep 20, 16:26 BST
Update - This issue has just cleared and nodata alerts are active again. We will continue to look into the root cause of this issue for another 24h.
Sep 19, 18:41 BST
Update - We are observing a number of received device payloads lower than normal again and nodata alerts are disabled at the moment.
Sep 19, 17:45 BST
Update - We have been observing the expected 5min wait time on nodata alerts for the past 3 hours, which means that received payloads have normalized. We will keep monitoring this issue for the next 24 hours.
Sep 18, 17:10 BST
Monitoring - We are observing a number of received device payloads lower than normal. The drop has been fluctuating between 1.5% and 2.5%. Bellow 2% our protection for false "no data" alerts triggers and delays these alerts delivery. So, instead of the builtin 5min wait we are seeing up to 15min wait.
We are continuing to monitor this issue and will update the status in about 3 hours. If it remains we'll proceed to adjust our protection threshold.
Sep 18, 12:23 BST