On October 5th between 14:20 and 14:50 our api.serverdensity.io/alerts was unavailable to requests.

This was caused by a networking failure at our provider that prevented traffic to be routed to the impacted service. Server Density is comprised of multiple micro-services, each of those behind a load balancing device pair sharing a virtual IP address. Traffic to this virtual address is guaranteed by our provider networking infrastructure.

While we are still working with our provider to help them find the root cause of this particular failure, we have reduced our alerting wait time so we are faster to manually re-route traffic in case of a re-occurrence. We will share the conclusions of that analysis as soon as they become available.

Also, we are currently close to completing a project that will change how this traffic is routed to each service that will remove the virtual IP address routing dependency.