Presented By O’Reilly and Cloudera
Make Data Work
September 11, 2018: Training & Tutorials
September 12–13, 2018: Keynotes & Sessions
New York, NY

Kafka at PayPal: Enabling 400 billion messages a day

Kevin Lu (PayPal), MAULIN VASAVADA (PayPal), Na Yang (PayPal)
3:30pm–4:10pm Thursday, 09/13/2018
Streaming systems & real-time applications
Location: 1E 09 Level: Intermediate
Secondary topics:  Data Integration and Data Pipelines, Data Platforms, Financial Services
Average rating: ****.
(4.00, 3 ratings)

Who is this presentation for?

  • Anyone interested in running Kafka at scale

Prerequisite knowledge

  • Basic knowledge of Kafka or another messaging system

What you'll learn

  • Understand the criticality of a large-scale Kafka environment, the set of tools you'll need to make this environment work and supportable, and how to provide the performance and scalability needed for PayPal's data volume and SLA

Description

Kafka is playing an increasingly important role in messaging and streaming systems. Managing fast-growing Kafka deployments and supporting customers with various requirements can become a challenging task for a small team of only a few engineers.

The availability of the Kafka infrastructure is essential to PayPal’s revenue stream. The company needs to catch issues before systems break down, know exactly how available it is for each client, and preemptively recover from problems when they occur. It also needs to have a clear view of message loss in its end-to-end Kafka pipeline. Operational tooling is critical to PayPal’s success, and the company has developed tools such as data loss auditing, full and partial cluster failovers, client and server-side KPI measurements, and a control panel for Kafka clusters.

Kevin Lu, Maulin Vasavada, and Na Yang explore the management and monitoring PayPal applies to Kafka, from client-perceived statistics to configuration management, failover, and data loss auditing. You’ll discover the criticality of a large-scale Kafka environment, the set of tools you’ll need to make this environment work and supportable, and how to provide the performance and scalability needed for PayPal’s data volume and SLA. Along the way, they highlight the architecture of PayPal’s next-generation Kafka monitoring and management system, built for serving all the Kafka-as-a-service needs.

Photo of Kevin Lu

Kevin Lu

PayPal

Kevin Lu is a software engineer at PayPal developing various Kafka components. He holds a degree in computer science from the University of California, Berkeley. Kevin first discovered his passion for coding in high school, when he developed plug-ins for Minecraft.

Photo of MAULIN VASAVADA

MAULIN VASAVADA

PayPal

Maulin Vasavada is a software developer and an architect on the Kafka team at PayPal, building a suite of components for Kafka as a service. He has strong experience building large-scale financial systems, shipping and logistics software, and software release management systems. Previously, he worked for eBay and as a consultant for Sun Microsystems.

Photo of Na Yang

Na Yang

PayPal

Na Yang is a software engineer at PayPal, where she focuses on building a scalable streaming infrastructure platform. Previously, she built various big data and distributed systems at MapR and Quova. Outside of work, she likes to spend time hiking with her kids.

Comments on this page are now closed.

Comments

Thomas Roach | SYSTEMS ENGINEER
09/17/2018 6:33am EDT

I’m still looking for slides. Please let us know where we can get them at. Thanks.

Picture of MAULIN VASAVADA
MAULIN VASAVADA | SOFTWARE DEVELOPER
09/15/2018 1:38pm EDT

Yes it should be

Picture of Matt Bolte
Matt Bolte | PRINCIPAL SOFTWARE ENGINEER
09/15/2018 11:57am EDT

Will the slides be made available?