Using machine learning to simplify Kafka operations

Description

Apache Kafka—an open source stream processing platform that provides a unified, high-throughput, low-latency methodology for handling real-time data feeds—is widely integrated into enterprise-level infrastructures for a variety of use cases, such as ingesting data into clusters, live data feeds to enable real-time processing, and replicating input data across clusters and data centers to provide 24×7 application uptime. While Kafka’s interface is delightfully simple, operating Kafka clusters is a challenge. It’s hard to troubleshoot misbehaving Kafka clusters. The root cause of why a real-time application is lagging may be due to an application problem (e.g., poor data partitioning or load imbalance) or due to a Kafka problem (e.g., resource exhaustion or suboptimal configuration).

All this is to say that getting the best performance, predictability, and reliability for Kafka-based applications is a complex art. Shivnath Babu and Dhruv Goel explain how to simplify the process by leveraging recent advances in machine learning and AI and outline a methodology for applying statistical learning to the rich and diverse monitoring data that is available from Kafka. This monitoring data includes metrics from Kafka brokers, producers, consumers, and the infrastructure as well as logs from various components of the Kafka ecosystem. Shivnath and Dhruv discuss how to automatically identify the root cause for a number of Kafka-based application bottlenecks, slowdowns, and failures using improved algorithms for anomaly detection, correlation, and forecasting and how the models they trained from production Kafka clusters have enabled effective capacity planning that has led to lower costs and higher reliability compared to the guesstimates done previously.