Automation of root cause analysis for big data stack applications

Description

In multiple distributed systems, applications can fail due to many reasons, such as out-of-memory or due to a timeout while waiting for some resource. Or the root cause may be deeper. For example, a timeout may be due to an application getting delayed because it accesses tables containing small files or non-splittable files and thus, accessing data on them is particularly slow. However the reason might be, when an application fails, users are required to fix the cause of the failure to get the application running successfully. Since applications may interact with multiple components, a failed application can generate a large set of raw logs. These logs typically contain thousands of messages, including errors and stacktraces. Hunting for the root cause of an application failure from these messy, raw, and distributed logs is hard for experts, and a nightmare for the thousands of new users coming to the big data stack.

Alkis Simitsis and Shivnath Babu share an automated technique for root cause analysis (RCA) for big data stack applications using deep learning techniques, using Spark and Impala. They begin by describing how to automatically generate insights into a failed application in a multiengine big data stack before detailing their approach to automatically identify the root cause of application failure, which consists of continuous log collection of Spark and Impala application failures and an automatic labeling mechanism using unsupervised learning; converting logs into feature vectors using a three-layer neural network; and learning a predictive model for RCA from these feature vectors using deep learning and active learning techniques. They conclude by discussing algorithms for automatic fixes for failed applications that use examples of successful and failed runs of the application or similar applications from history. They’ll then try out a limited number of alternative configurations to get the application quickly to a running state and walk you through getting the application to a resource-efficient running state.