Feb 3, 12:22 GMT
72 hours have passed since we observed the last occurrence of this incident and since we replaced the high latency server. We are considering this resolved and will be publishing a post-mortem in the next few days.
Feb 1, 18:01 GMT
We had to replace the high latency cluster server. We're now monitoring if the replacement is free from that behavior.
Jan 29, 21:55 GMT
We're continuing to monitor this we saw a re-occurrence of the high latency earlier today which prevented us from re-adding the removed cluster member. We are working with our provider to solve this issue.
All systems are operational and there has not been new impact to service since yesterday 16:30 UTC
Jan 28, 21:31 GMT
We observed high network latency to one member of this cluster. This member has now been removed, restoring normal cluster function. We'll continue to monitor this situation.
Jan 27, 17:02 GMT
We are still keeping this incident open because we just had a stall on that same cluster that could have caused some device no data alerts to trigger.
Jan 27, 16:33 GMT
We have now finished recovering this cluster and we'll be monitoring its health for the next hours.
Jan 27, 13:57 GMT
We're still working on the recovery of the affected cluster and will share updates shortly.
Jan 27, 13:35 GMT
We are currently experiencing a failure in one of our distributed synchronization clusters which is preventing us from serving UI or API requests. Work is ongoing to recover that cluster.
Jan 27, 13:07 GMT