Engineering the Future of Software
Feb 25–26, 2018: Training
Feb 26–28, 2018: Tutorials & Conference
New York, NY

Apache Kafka + Apache Mesos = Highly scalable streaming microservices

Kai Wähner (Confluent)
4:50pm–5:40pm Tuesday, February 27, 2018
Secondary topics:  Framework-focused, Hands-on
Average rating: ****.
(4.00, 2 ratings)

Who is this presentation for?

  • Architects, consultants, developers, and project leads

Prerequisite knowledge

  • Familiarity with distributed systems and containers

What you'll learn

  • Explore a highly scalable, mission-critical infrastructure using Apache Kafka and Apache Mesos


Microservices establish many benefits like Agile, flexible development, and deployment of business logic. However, a microservice architecture also creates many new challenges, including increased communication between distributed instances, the need for orchestration, new failover requirements, and resiliency design patterns.

Kai Wähner offers an overview of a highly scalable, mission-critical infrastructure using Apache Kafka and Apache Mesos: Kafka brokers are used as the distributed messaging backbone; Kafka’s Streams API embeds stream processing into any external application without the need for a dedicated streaming cluster; and Mesos is used as a scalable infrastructure to leverage the benefits of a cloud-native platform like service discovery, health checks, and failover management. Kai ends with a live demo that shows how to develop real-time applications for your core business with Kafka messaging brokers and Kafka Streams API and how to deploy, manage, and scale them on a Mesos cluster using different deployment options.

Photo of Kai Wähner

Kai Wähner


Kai Wähner is a technology evangelist at Confluent. Kai’s areas of expertise include big data analytics, machine learning, deep learning, messaging, integration, microservices, the internet of things, stream processing, and blockchain. He’s regular speaker at international conferences such as JavaOne, O’Reilly Software Architecture, and ApacheCon and has written a number of articles for professional journals. Kai also shares his experiences with new technologies on his blog.