To gain insights from large-scale time series metrics and use them as the basis for accurate predictions, root cause diagnosis, and other tasks, it’s important to discover the relationships among the metrics (i.e., the correlations between them). For example, if you need to predict how much revenue an ecommerce site will generate this quarter, one very rough method would be to use the previous quarter’s revenue as a guide, but it will not take into consideration any other valid parameters, such as how much traffic came to the site in the current quarter, the site’s bounce rate, or other metrics that may be much better predictors.
However, to understand which metrics can be used as predictors (or other tasks), one must understand what metrics are related to each other and how. For a small-scale operation, these relationships can be manually defined. For certain types of metrics, such as IT, tools such as configuration management databases (CMDB) may automate some of the discovery of the relationships between the metrics. But if you want to incorporate metrics beyond IT (e.g., application metrics or business metrics like revenue) at the vast scale most digital businesses require, machine learning tools are needed.
Ira Cohen shares key machine-learning methods for correlating metrics at scale, without having to do any manual configuration. Implementing these methods at scale can be computationally expensive, so Ira suggests methods for reducing the computational resources needed. (In particular, Ira explains how to efficiently scale the similarity and clustering methods.) And since correlation does not necessarily equal causation, Ira also covers ways to identify causality.
Ira Cohen is a cofounder and chief data scientist at Anodot, where he’s responsible for developing and inventing the company’s real-time multivariate anomaly detection algorithms that work with millions of time series signals. He holds a PhD in machine learning from the University of Illinois at Urbana-Champaign and has over 12 years of industry experience.
Comments on this page are now closed.
©2017, O’Reilly UK Ltd • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org