Kafka incurs significant management overhead. Growing cluster sizes, the increasing volume and diversity of user traffic, and the age of network and server components further contribute to this overhead. The resulting increase in the frequency of hardware failures and load imbalance leads to frequent service interruptions, leading to poor user experience. In particular, reactive mitigation becomes insufficient due to the impact on the other services that have a Kafka dependency. Getting near-optimal performance from such an infrastructure service, maintaining its availability in the face of cascading failures, and achieving these objectives with minimal management overhead are critical but nontrivial tasks.
Adem Efe Gencer explains how LinkedIn alleviated the management overhead of large-scale Kafka clusters using Cruise Control. Adam begins by outlining Cruise Control’s approach to monitoring load distribution in clusters, identifying an imbalance in them, and fixing this imbalance using replica and leadership movements. He then explains how Cruise Control detects fail-stop broker failures and SLO violations without human intervention and examines a more aggressive scenario, where Cruise Control proactively identifies and mitigates potential service disruptions.
Adem Efe Gencer develops Apache Kafka and the ecosystem around it and supports their operation at LinkedIn. In particular, he works on the design, development, and maintenance of Cruise Control, a system for alleviating the management overhead of large-scale Kafka clusters. He actively acts as a reviewer for top-tier journals and conferences. He holds a PhD in computer science from Cornell University, where his research has focused on improving the scalability of blockchain technologies. The protocols introduced in his research were adopted by Waves Platform, Aeternity, Cypherium, Enecuum, Ergo Platform, and Legalthings and are actively being developed into other systems. His papers been cited over 500 times. He received a best student paper award in Middleware Conference.
©2019, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com