All Systems Operational
Alerting ? Operational
Alert Delivery Operational
SMS Operational
E-mail Operational
PagerDuty (Incident Creation) Operational
PagerDuty (Notification Delivery) Operational
Slack Operational
Webhooks Operational
HipChat Operational
Push notifications (global) ? Operational
Push notifications (iOS) Operational
Push notifications (Android) Operational
Agent payloads ? Operational
API Operational
Availability monitoring ? Operational
Web UI Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Aug 18, 2017

No incidents reported today.

Aug 17, 2017

No incidents reported.

Aug 16, 2017

No incidents reported.

Aug 15, 2017

No incidents reported.

Aug 14, 2017

No incidents reported.

Aug 13, 2017

No incidents reported.

Aug 12, 2017

No incidents reported.

Aug 11, 2017

No incidents reported.

Aug 10, 2017

No incidents reported.

Aug 9, 2017

No incidents reported.

Aug 8, 2017
Postmortem - Read details
Aug 8, 14:00 BST
Resolved - This incident has been resolved.
Aug 8, 13:58 BST
Monitoring - Today we have confirmed the source of the request spike causing this incident.
After today we don't expect a re-occurrence and we have moved the incident state to "Monitoring" while we work with the source of these requests to remove it's negative impact.
Jul 11, 08:19 BST
Update - During there last occurrences we narrowed down the cause of the request spike as coming from our api (api.serverdensity.io) and not from eg. the user facing app or incoming device payloads.
Today we were able to prevent the daily 07:00 UTC occurrence by blocking a set of suspect API calls. This has reduced the issue scope even further, putting us closer to a solution. Today's impact was a 4 minute unavailability (06:58 - 07:02) on that set of API calls.
Jul 9, 08:15 BST
Update - We have kept this incident open this long as this is an event only happening at 07:00 UTC, preventing us from continuously verifying possible corrections. We are continuing to work on it.
We'll update this again tomorrow after 07:00 UTC.
Jul 8, 08:37 BST
Update - Between 07:00 and 07:11 UTC we had a re-occurrence of this incident. The consequence was immediately mitigated but we are still following up on the root cause of this data request spike.
Jul 7, 08:24 BST
Update - Payload processing is normal since 08:15 UTC. We're continuing to work on the cause the observed request spike.
Jul 6, 12:02 BST
Identified - We have identified a reduction in our device payload processing capacity caused by an abnormal data request. This may show on some devices as missing metrics data. Alerting is not affected.
We've adjusted capacity while we identify and resolve the request spike.
Jul 6, 08:49 BST
Aug 7, 2017

No incidents reported.

Aug 6, 2017

No incidents reported.

Aug 5, 2017

No incidents reported.

Aug 4, 2017

No incidents reported.