We have not seen a recurrence of this issue in over 24h after deploying a code fix. So we're considering this resolved and will publish a detailed postmortem in a few days.
Posted Feb 08, 2017 - 16:50 GMT
Monitoring
Graphs loading is now normalized. A gap may be visible between 12:48 and 13:05 UTC. This will even out in the next few days.
We are continuing to monitor for re-occurences closely while we work on the root cause.
Posted Feb 06, 2017 - 13:16 GMT
Identified
This is happening again now and we are working to restore graphs.
Posted Feb 06, 2017 - 13:03 GMT
Monitoring
We have normalized all redundancy after confirming the replacement members are working within the expected parameters.
This will now be monitored closer for the next few hours.
Posted Feb 05, 2017 - 23:08 GMT
Update
The slowest members have been removed and graphs should be loading faster now as well as gaps should be gone. We're continuing to work on this to normalize redundancy.
Posted Feb 05, 2017 - 22:20 GMT
Identified
We have identified a slow member in one of our metrics clusters which powers graphs and are removing it from rotation.
Some graphs gaps may be showing.