Bring the Noise: Making Effective Use of a Quarter Million Metrics

At Etsy, we collect over a quarter million metrics from a variety of monitoring systems – everything from 404s to how much money we make, all in real time. However, with so many metrics and only a hundred engineers, how can we effectively monitor everything to separate the signal from the noise?

We’d like to introduce Velocity to two complementary tools we’ve developed to solve this problem.

Skyline is our real-time anomaly detection system. It continually analyzes each of our metrics, as they come in, for anomalous behavior. The algorithm we use automatically determines what it means for any given metric to be anomalous, and when an anomaly is detected, our engineers are alerted via interactive dashboard and can react accordingly.

Of course, with so many metrics, an anomalous event often impacts many metrics in similar ways. We wanted to surface these correlations automatically, to avoid manually curating dashboards, so we built Oculus – a way to index and compare all metrics with each other for similarity. This way, we can detect how a problem impacts many different metrics at once, furthering our understanding of the incident. We can then save and index the incident “pattern” itself so that if it happens again, Oculus will let us know immediately, while telling us the past diagnosis and solution.

This joint talk will cover the following topics:

• Metric Overload – The situation which gave rise to the development of these tools and how we approached the problem
• Skyline – The architecture and algorithms we use for realtime anomaly detection on a massive scale
• Oculus – The architecture and algorithms we use to compute similarity and correlation across all of our metric data
• Demonstration – This talk wouldn’t be any fun if we didn’t show you the tools firsthand!