Presented By O’Reilly and Cloudera
Make Data Work
March 5–6, 2018: Training
March 6–8, 2018: Tutorials & Conference
San Jose, CA

Kafka streaming applications with Akka Streams and Kafka Streams

Dean Wampler (Anyscale)
11:00am11:40am Thursday, March 8, 2018
Secondary topics:  Expo Hall
Average rating: *****
(5.00, 1 rating)

Who is this presentation for?

  • Data engineers and architects

Prerequisite knowledge

  • A working knowledge of Kafka and streaming (useful but not required)

What you'll learn

  • Understand how to combine Kafka with Akka Streams and Kafka Streams to implement various streaming scenarios that leverage the strengths of these tools while avoiding their weaknesses and how they compare to Spark Streaming and Flink


Kafka Streams is purpose-built for reading data from Kafka topics, processing it, and writing the results to new topics. With powerful stream and table abstractions and an exactly once capability, it supports a variety of common scenarios involving transformation, filtering, and aggregation. Akka Streams emerged as a dataflow-centric abstraction for the Akka Actors model, designed for general-purpose microservices, especially when per-event low-latency is important. Most systems provide efficient processing amortized over sets of records, but usually not at end-to-end low latency per event (e.g., for complex event processing in true real-time applications). Also because of its general-purpose nature, Akka Streams supports a wider class of application problems and third-party integrations but is less focused on Kafka-based applications. Both are primarily libraries that you integrate into your microservices, which means you must manage their lifecycles yourself, but you also get lots of flexibility to do this as you see fit.

In contrast, Spark Streaming and Flink run their own services. You write “jobs” or use interactive shells that tell these services what computations to do over data sources and where to send results. Spark and Flink then determine what processes to run in your cluster to implement the dataflows. Hence, there is less of a DevOps burden to bear but also less flexibility when you might need it. Both systems are also more focused on data analytics problems, with various levels of support for SQL over streams, machine learning model training and scoring, etc.

Dean Wampler compares and contrasts data processing with Akka Streams and Kafka Streams, discussing the strengths and weaknesses of each tool for particular design needs and contrasting them with Spark Streaming and Flink, so you’ll know when to choose them instead.

Photo of Dean Wampler

Dean Wampler


Dean Wampler is an expert in streaming data systems, focusing on applications of machine learning and artificial intelligence (ML/AI). He’s head of developer relations at Anyscale, which is developing Ray for distributed Python, primarily for ML/AI. Previously, he was an engineering VP at Lightbend, where he led the development of Lightbend CloudFlow, an integrated system for building and running streaming data applications with Akka Streams, Apache Spark, Apache Flink, and Apache Kafka. Dean is the author of Fast Data Architectures for Streaming Applications, Programming Scala, and Functional Programming for Java Developers, and he’s the coauthor of Programming Hive, all from O’Reilly. He’s a contributor to several open source projects. A frequent conference speaker and tutorial teacher, he’s also the co-organizer of several conferences around the world and several user groups in Chicago. He earned his PhD in physics from the University of Washington.