Big data has been playing a vital role in every sphere of business, for example surfacing personalized content in timelines, to provide highly available and performant service to the end user. This rests, in part, upon the availability of high fidelity data. However, exogenic and/or endogenic factors often give rise to anomalies. At web scale, with a large number of services and with each service having a large set of metrics, visual detection of anomalies is not pragmatic. Furthermore, automatic detection of anomalies is non-trivial owing to the following reasons:
To this end, at Twitter we developed novel statistical techniques for automatically detecting anomalies. In January 2015, we open-sourced a standalone R package. Since then, we have extended the techniques to leverage multiple time series to minimize the false positives rate. Specifically:
1. We exploit correlations between metrics – for example, multiple metrics of the different components in a Storm topology.
2. We address the potential skew between different time series via interval intersection and/or convolution analysis.
The techniques we shall present were evaluated with a wide variety – system and application metrics obtained from production – of time series.
The proposed talk is complementary to the talk I gave at Velocity in November 2014.
@arun_kejariwal is a software engineer at Twitter, where he works on research and development of novel techniques for time series analysis. Prior to joining Twitter, Arun worked on research and development of practical and statistically rigorous methodologies to deliver high performance, availability, and scalability in large-scale distributed clusters. Some of the techniques he helped develop have been published in peer-reviewed international conferences and journals.
Arun received his Bachelor’s degree in EE from IIT Delhi and doctorate in CS from UCI.
Karthik is the engineering manager and technical lead for Real Time Analytics at Twitter. He has two decades of experience working in parallel databases, big data infrastructure and networking. He cofounded Locomatix, a company that specializes in real timestreaming processing on Hadoop and Cassandra using SQL that was acquired by Twitter. Before Locomatix, he had a brief stint with Greenplum where he worked on parallel query scheduling. Greenplum was eventually acquired by EMC for more than $300M. Prior to Greenplum, Karthik was at Juniper Networks where he designed and delivered platforms, protocols, databases and high availability solutions for network routers that are widely deployed in the Internet. Before joining Juniper at University of Wisconsin, he worked extensively in parallel database systems, query processing, scale out technologies, storage engine and online analytical systems. Several of these research were spun as a company later acquired by Teradata.
He is the author of several publications, patents and one of the best selling book “Network Routing: Algorithms, Protocols and Architectures.” He has a Ph.D. in Computer Science from UW Madison with a focus on databases.
Comments on this page are now closed.