Build Systems that Drive Business
Sep 30–Oct 1, 2018: Training
Oct 1–3, 2018: Tutorials & Conference
New York, NY

Kafka Streams in practice: What works and what doesn’t (yet)

Bart De Vylder (CoScale)
3:50pm–4:30pm Tuesday, October 2, 2018
Distributed Data, Distributed Systems
 Location: Nassau Level: Beginner
Secondary topics:  Resilient, Performant & Secure Distributed Systems
Average rating: ****.
(4.00, 2 ratings)

What you'll learn

  • Explore Kafka Streams and the strategies and challenges involved in migrating an existing codebase and production environment to a streaming architecture

Description

Driven by a need for scalability, CoScale recently converted part of its backend to a streaming architecture using Kafka Streams, an open source library for writing streaming applications, which ships with Kafka. Kafka Streams is a relatively new addition to available streaming frameworks (Storm, Samza, Flink, Spark Streaming, etc.), but it has some unique features that make a nice fit for the challenges faced.

Bart De Vylder makes the case for Kafka Streams by comparing the main design decisions to other streaming frameworks. Along the way, he shares his experience converting an existing codebase to Kafka Streams and migrating an existing production environment, highlighting aspects that worked remarkably well as well as the challenges he ran into and the workarounds he used to solve them.

Photo of Bart De Vylder

Bart De Vylder

CoScale

Bart De Vylder is a data scientist at CoScale. Previously, Bart was active in software engineering and architecture, with a focus on distributed systems. His interests lie in machine learning and building reliable, scalable data processing systems. Bart holds a PhD in artificial intelligence from the Free University of Brussels.