Gap in graphs

Incident Report for Server Density

Postmortem

On October 4 between 18:46 and 18:54 and between 19:17 and 20:00 UTC no device metrics were recorded.

This happened during a maintenance procedure on one of our database clusters when a cluster member switched to a state not permitting operations.

We use a MongoDB replica set for this cluster and the procedure being executed required the removal of a member that was to be decomissioned. The order in which the procedure was prepared was incorrect as the member needs to be shutdown before modifying the replica set configuration to logically remove the member afterwards.

We have corrected the procedure and confirmed the expected behavior both ourselves in our environments and with the vendor.

Posted Oct 10, 2017 - 13:04 BST

Resolved

This incident has been resolved.

Posted Oct 04, 2017 - 21:35 BST

Monitoring

This has been identified as having been caused by a state transition on a secondary database server undergoing maintenance. There was a second occurrence between 19:17 and 20:00 UTC while we completely removed the affected server.
We'll be monitoring that database cluster for another hour but are not expecting new occurrences.

Posted Oct 04, 2017 - 20:35 BST

Investigating

We are investigating a gap showing in graphs between 18:46 and 18:54 UTC.
Alerting is not affected.

Posted Oct 04, 2017 - 19:59 BST