Kafka is playing an increasingly important role in messaging and streaming systems. Managing fast-growing Kafka deployments and supporting customers with various requirements can become a challenging task for a small team of only a few engineers.
The availability of the Kafka infrastructure is essential to PayPal’s revenue stream. The company needs to catch issues before systems break down, know exactly how available it is for each client, and preemptively recover from problems when they occur. It also needs to have a clear view of message loss in its end-to-end Kafka pipeline. Operational tooling is critical to PayPal’s success, and the company has developed tools such as data loss auditing, full and partial cluster failovers, client and server-side KPI measurements, and a control panel for Kafka clusters.
Kevin Lu, Maulin Vasavada, and Na Yang explore the management and monitoring PayPal applies to Kafka, from client-perceived statistics to configuration management, failover, and data loss auditing. You’ll discover the criticality of a large-scale Kafka environment, the set of tools you’ll need to make this environment work and supportable, and how to provide the performance and scalability needed for PayPal’s data volume and SLA. Along the way, they highlight the architecture of PayPal’s next-generation Kafka monitoring and management system, built for serving all the Kafka-as-a-service needs.
Kevin Lu is a software engineer at PayPal developing various Kafka components. He holds a degree in computer science from the University of California, Berkeley. Kevin first discovered his passion for coding in high school, when he developed plug-ins for Minecraft.
Maulin Vasavada is a software developer and an architect on the Kafka team at PayPal, building a suite of components for Kafka as a service. He has strong experience building large-scale financial systems, shipping and logistics software, and software release management systems. Previously, he worked for eBay and as a consultant for Sun Microsystems.
Na Yang is a software engineer at PayPal, where she focuses on building a scalable streaming infrastructure platform. Previously, she built various big data and distributed systems at MapR and Quova. Outside of work, she likes to spend time hiking with her kids.
Comments on this page are now closed.
For exhibition and sponsorship opportunities, email strataconf@oreilly.com
For information on trade opportunities with O'Reilly conferences, email partners@oreilly.com
View a complete list of Strata Data Conference contacts
©2018, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • confreg@oreilly.com
Comments
I’m still looking for slides. Please let us know where we can get them at. Thanks.
Yes it should be
Will the slides be made available?