Real-time anomaly detection on observability data using neural networks

Description

Expedia is on a mission of connecting people to places through the power of technology. To accomplish this, the company builds and runs hundreds of microservices that provide different functionalities to serve one single customer request.
But there’s still the possibility of one or more services failing at the same time. You’ll take a look at how Expedia determines these failed services in automated manner and provides a high quality of service, which has led to huge improvements in the company’s mean time to know (MTTK) and mean time to resolve (MTTR).

Keshav Peswani and Ashish Aggarwal take you on the journey of distributed tracing in Expedia that started with Zipkin as a prototype and ended with the company building its own solution (in open source) using OpenTracing APIs. You’ll take a deep dive into the architecture and see how Expedia ingests terabytes of tracing data in production for hundreds of microservices, and and how it uses this data for trending service errors, latencies, and rate. With the increasing number of microservices, there was the need to have a real-time, intelligent alerting and monitoring system to contribute to the goal of reducing MTTK and MTTR and move toward 24-7 reliability.

With unique behavioral patterns for each of the service errors, leveraging neural networks to understand the behavior changes for each of the microservice and raise alerts was indeed a challenging task. The task uncovered a few unexpected challenges, and the solution was less straightforward than initially estimated. But ultimately the anomaly detector using a neural network produced results that beat the company’s expectations, once again validating the interest in neurocomputing that’s overtaking the industry.

To achieve this, Keshav and Ashish predict the service failures in the microservices using recurrent neural networks on trends data and perform anomaly detection on predicted values. They demonstrate how to train a recurrent neural network and autotune hyperparameters using Bayesian optimization methods. And they detail the architecture for the automated training pipeline using AWS SageMaker and Lambda, as well as how the anomaly detection works in a streaming manner using Kafka (KStreams) as the backbone and model deployed on SageMaker in a cost-effective manner. Currently, Expedia plans to take human intervention into consideration to refine the alerts in order to reduce false positives and form a five-step methodology for anomaly detection.