Big data has been playing a vital role in every sphere of business, for example surfacing personalized content in timelines, to provide highly available and performant service to the end user. This rests, in part, upon the availability of high fidelity data. However, exogenic and/or endogenic factors often give rise to anomalies. At web scale, with a large number of services and with each service having a large set of metrics, visual detection of anomalies is not pragmatic. Furthermore, automatic detection of anomalies is non-trivial owing to the following reasons:
To this end, at Twitter we developed novel statistical techniques for automatically detecting anomalies. In January 2015, we open-sourced a standalone R package. Since then, we have extended the techniques to leverage multiple time series to minimize the false positives rate. Specifically:
1. We exploit correlations between metrics – for example, multiple metrics of the different components in a Storm topology.
2. We address the potential skew between different time series via interval intersection and/or convolution analysis.
The techniques we shall present were evaluated with a wide variety – system and application metrics obtained from production – of time series.
The proposed talk is complementary to the talk I gave at Velocity in November 2014.
@arun_kejariwal is a software engineer at Twitter, where he works on research and development of novel techniques for time series analysis. Prior to joining Twitter, Arun worked on research and development of practical and statistically rigorous methodologies to deliver high performance, availability, and scalability in large-scale distributed clusters. Some of the techniques he helped develop have been published in peer-reviewed international conferences and journals.
Arun received his Bachelor’s degree in EE from IIT Delhi and doctorate in CS from UCI.
Karthik Ramasamy is the cofounder of Streamlio, a company building next-generation real-time processing engines. Karthik has more than two decades of experience working in parallel databases, big data infrastructure, and networking. Previously, he was engineering manager and technical lead for real-time analytics at Twitter, where he was the cocreator of Heron; cofounded Locomatix, a company that specialized in real-time stream processing on Hadoop and Cassandra using SQL (acquired by Twitter); briefly worked on parallel query scheduling at Greenplum (acquired by EMC for more than $300M); and designed and delivered platforms, protocols, databases, and high-availability solutions for network routers at Juniper Networks. He is the author of several patents, publications, and one best-selling book, Network Routing: Algorithms, Protocols, and Architectures. Karthik holds a PhD in computer science from the University of Wisconsin-Madison with a focus on databases, where he worked extensively in parallel database systems, query processing, scale-out technologies, storage engines, and online analytical systems. Several of these research projects were spun out as a company later acquired by Teradata.
Comments on this page are now closed.