The secret sauce behind LinkedIn's self-managing Kafka clusters

Description

Kafka clusters should be run with rack awareness, a balanced utilization of all the hardware resources (CPU, disk, and network), and automatic recovery from failures. When running small clusters, the above requirements can be satisfied manually (e.g., by SREs monitoring the broker resource distribution and manually reassigning replicas from hot brokers to less loaded brokers). However, doing so becomes prohibitively expensive with a large Kafka deployment. LinkedIn runs more than 1,800+ Kafka brokers that deliver more than two trillion messages a day. Running Kafka at such a scale makes automated operations a necessity, and LinkedIn has been exploring ways to do so.

There are a few challenges to be addressed:

• Monitoring the performance at a fine granularity: This is trickier than it sounds. For example, how can we decide the CPU utilization of a replica while only CPU utilization of a broker is available?
• Multipurpose optimization: Because there are multiple requirements to be met, it is complicated to take all the goals into consideration. Solving such multipurpose optimization can take an extremely long time and render the approach infeasible.
• Avoiding any impact on availability: It is important to ensure that normal traffic is not affected while optimizing a Kafka cluster. We may also wish to interrupt some long running optimizations if necessary.

Jiangjie Qin explains how LinkedIn addressed the above problems and shares lessons learned from operating Kafka at scale with minimum human intervention. This experience can also be applied to the management of other stateful distributed systems. While there are existing products that help balance resource utilization in a distributed system, most of these are application agnostic and perform the rebalance by migrating the entire application process. While this works well for stateless systems, it typically falls short when it comes to stateful systems (e.g., Kafka) due to the large amount of state associated with the process. LinkedIn’s solution addresses this problem by trying to understand the application and migrating only a partial state—a solution that could be useful in any stateful distributed system.