Lightning-fast time series modeling and prediction: (S)ARIMA on steroids

Description

For decades, researchers and practitioners have used different statistical techniques for modeling time-indexed data (i.e., time series). One model among many is still under the spotlight: ARIMA.

The ARIMA family model is the best time series model in some circumstances. In practice, it performs very well at predicting the future of a time series and even against highly sophisticated super-new models like long short-term memory (LSTM), according to benchmarks. Moreover, a complete theoretical analysis was conducted in the last decades. So it’s a rare machine learning technique where a mathematical analysis is tractable (but not simple, unfortunately). For all those reasons, ARIMA is a central element in the time series analysis field. With the growing user base, a subproblem arose that remained unsolved for decades: the efficient estimation of seasonal ARIMA models a.k.a. (S)ARIMA.

Meir Toledano explains how Anodot solved this problem, because the seasonal problem is extremely important in practice. Most of the gathered data that monitors human activity is seasonal. For example, the number of taxi rides sampled hourly exhibits a weekly pattern: more rides in the day than in the night; more rides during the week than during the weekend. What’s true for taxi rides is true for electricity consumption, hotel reservations, car traffic, ad spending, ecommerce revenue, etc.

Using (S)ARIMA has been challenging. Standard implementations take around 15 minutes to learn the model, mainly because the algorithm has to “see” very far in past in order to “understand” what’s happening. This limitation makes (S)ARIMA impractical to use in the big data era. Anodot estimates it can now use (S)ARIMA for more than 100 million metrics every day. On average it takes 80 ms to estimate the model, a 5 orders of magnitude improvement! This was made possible from breakthroughs in theory and code implementation.

Meir demonstrates how to use the model to make predictions and for real-time anomaly detection, where Anodot is a specialist.