Bring the Noise: Making Effective Use of a Quarter Million Metrics

Operations and Culture
Location: Palace Suite - Blenheim Room
Average rating: ****.
(4.57, 44 ratings)

At Etsy, we collect over a quarter million metrics from a variety of monitoring systems – everything from 404s to how much money we make, all in real time. However, with so many metrics and only a hundred engineers, how can we effectively monitor everything to separate the signal from the noise?

We’d like to introduce Velocity to two complementary tools we’ve developed to solve this problem.

Skyline is our real-time anomaly detection system. It continually analyzes each of our metrics, as they come in, for anomalous behavior. The algorithm we use automatically determines what it means for any given metric to be anomalous, and when an anomaly is detected, our engineers are alerted via interactive dashboard and can react accordingly.

Of course, with so many metrics, an anomalous event often impacts many metrics in similar ways. We wanted to surface these correlations automatically, to avoid manually curating dashboards, so we built Oculus – a way to index and compare all metrics with each other for similarity. This way, we can detect how a problem impacts many different metrics at once, furthering our understanding of the incident. We can then save and index the incident “pattern” itself so that if it happens again, Oculus will let us know immediately, while telling us the past diagnosis and solution.

This joint talk will cover the following topics:

  • Metric Overload – The situation which gave rise to the development of these tools and how we approached the problem
  • Skyline – The architecture and algorithms we use for realtime anomaly detection on a massive scale
  • Oculus – The architecture and algorithms we use to compute similarity and correlation across all of our metric data
  • Demonstration – This talk wouldn’t be any fun if we didn’t show you the tools firsthand!
Photo of Jon Cowie

Jon Cowie

Chef

Jon Cowie is a Principal Customer Architect at Chef where he’s working on solving interesting technical problems and interesting people problems. In his former life as an individual contributor he wrote O’Reilly’s “Customizing Chef” book and contributed to a number of open source projects.

Sponsors

Sponsorship Opportunities

For exhibition and sponsorship opportunities at Velocity conference, contact Gloria Lombardo at +1 (203) 381-9245 or glombardo@oreilly.com

Media Partner Opportunities

For information on trade opportunities with O'Reilly conferences contact Jaimey Walking Bear at mediapartners
@oreilly.com

Contact Us

View a complete list of Velocity Europe 2013 contacts