Big data has been playing a vital role in every sphere of business, for example surfacing personalized content in timelines, to provide highly available and performant service to the end user. This rests, in part, upon the availability of high fidelity data. However, exogenic and/or endogenic factors often give rise to anomalies. At web scale, with a large number of services and with each service having a large set of metrics, visual detection of anomalies is not pragmatic. Furthermore, automatic detection of anomalies is non-trivial owing to the following reasons:
To this end, at Twitter we developed novel statistical techniques for automatically detecting anomalies. In January 2015, we open-sourced a standalone R package. Since then, we have extended the techniques to leverage multiple time series to minimize the false positives rate. Specifically:
1. We exploit correlations between metrics – for example, multiple metrics of the different components in a Storm topology.
2. We address the potential skew between different time series via interval intersection and/or convolution analysis.
The techniques we shall present were evaluated with a wide variety – system and application metrics obtained from production – of time series.
The proposed talk is complementary to the talk I gave at Velocity in November 2014.
@arun_kejariwal is a software engineer at Twitter, where he works on research and development of novel techniques for time series analysis. Prior to joining Twitter, Arun worked on research and development of practical and statistically rigorous methodologies to deliver high performance, availability, and scalability in large-scale distributed clusters. Some of the techniques he helped develop have been published in peer-reviewed international conferences and journals.
Arun received his Bachelor’s degree in EE from IIT Delhi and doctorate in CS from UCI.
Karthik Ramasamy is the co-founder of Streamlio that focuses on building next generation real time processing engines. Before Streamlio, he was the engineering manager and technical lead for real-time analytics at Twitter where he co-created Twitter Heron. He has two decades of experience working in parallel databases, big data infrastructure, and networking. He cofounded Locomatix, a company that specializes in real-time streaming processing on Hadoop and Cassandra using SQL, that was acquired by Twitter. Before Locomatix, he had a brief stint with Greenplum, where he worked on parallel query scheduling. Greenplum was eventually acquired by EMC for more than $300M. Prior to Greenplum, Karthik was at Juniper Networks, where he designed and delivered platforms, protocols, databases, and high availability solutions for network routers that are widely deployed on the internet. Before joining Juniper, at the University of Wisconsin he worked extensively in parallel database systems, query processing, scale out technologies, storage engines, and online analytical systems. Several of these research projects were later spun off as a company acquired by Teradata. Karthik is the author of several publications, patents, and Network Routing: Algorithms, Protocols and Architectures. He has a Ph.D. in computer science from the University of Wisconsin, Madison with a focus on big data and databases.
Comments on this page are now closed.