Sep 23–26, 2019
Please log in

Now you see me; now you compute: Building event-driven architectures with Apache Kafka

Michael Noll (Confluent)
1:15pm1:55pm Wednesday, September 25, 2019
Location: 1A 15/16
Average rating: *****
(5.00, 1 rating)

Who is this presentation for?

  • Architects, data engineers, application developers, data analysts, and data scientists




Would you cross the street with traffic information that’s a minute old? Certainly not. Modern businesses have the same needs nowadays, whether it’s due to competitive pressure or because their customers have much higher expectations of how they want to interact with a product or service. At the heart of this movement are events: in today’s digital age, events are everywhere. Every digital action—across online purchases to ride-sharing requests to bank deposits—creates a set of events around transaction amount, transaction time, user location, account balance, and much more. The technology that allows businesses to read, write, store, and compute and process these events in real-time are event-streaming platforms, and tens of thousands of companies like Netflix, Audi, PayPal, Airbnb, Uber, and Pinterest have picked Apache Kafka as the de facto choice to implement event-driven architectures and reshape their industries.

Michael Noll explores why and how you can use Apache Kafka and its growing ecosystem to build event-driven architectures that are elastic, scalable, robust, and fault tolerant, whether it’s on-premises, in the cloud, on bare metal machines, or in Kubernetes with Docker containers. Specifically, you’ll look at Kafka as the storage and publish and subscribe layer; Kafka’s Connect framework for integrating external data systems such as MySQL, Elastic, or S3 with Kafka; and Kafka’s Streams API and KSQL as the compute layer to implement event-driven applications and microservices in Java and Scala and streaming SQL, respectively, that process the events flowing through Kafka in real time. Michael provides an overview of the most relevant functionality, both current and upcoming, and shares best practices and typical use cases so you can tie it all together for your own needs.

Prerequisite knowledge

  • A basic understanding of Kafka (useful but not required)

What you'll learn

  • Learn how to turn your company into into real-time businesses and how to interact with customers in real-time by understanding the crucial role of business events (data) and their never-ending flow as the central driver for this transformation and by leveraging the foundation of Apache Kafka as the event streaming platform to capture, store, distribute, and process these business events
  • Discover how to navigate and apply the key components in Kafka's growing ecosystem so you're able to design and build your own end-to-end architectures in your organization
Photo of Michael Noll

Michael Noll


Michael Noll is the technologist of the office of the CTO at Confluent, the company founded by the creators of Apache Kafka. Previously, Michael was the technical lead of DNS operator Verisign’s big data platform, where he grew the Hadoop, Kafka, and Storm-based infrastructure from zero to petabyte-sized production clusters spanning multiple data centers—one of the largest big data infrastructures in Europe at the time. He’s a well-known tech blogger in the big data community. In his spare time, Michael serves as a technical reviewer for publishers such as Manning and is a frequent speaker at international conferences, including Strata, ApacheCon, and ACM SIGIR. Michael holds a PhD in computer science.

Comments on this page are now closed.


Picture of Michael Noll
Michael Noll | Technologist, Office of the CTO
10/02/2019 5:13pm EDT

@jesus cruz: Thanks for the kind words. Happy to hear you find it useful. The slides are available at

jesus cruz | PROJECT MANAGER
10/01/2019 3:17pm EDT

Hello, I attended your session during strata in NY las week. The presentation was totally perfect and I wonder if that presentation could be shared in order to take in count all the point you quoted in it since we are would like to desing and end-to-end architecture based on KAFKA in our organization.

  • Cloudera
  • O'Reilly
  • Google Cloud
  • IBM
  • Cisco
  • Dataiku
  • Intel
  • Io-Tahoe
  • MemSQL
  • Microsoft Azure
  • Oracle Cloud Infrastructure
  • SAS
  • Arcadia Data
  • BMC Software
  • Hazelcast
  • SAP
  • Amazon Web Services
  • Anaconda
  • Esri
  •, Inc.
  • Kyligence
  • Pitney Bowes
  • Talend
  • Google Cloud
  • Confluent
  • DataStax
  • Dremio
  • Immuta
  • Impetus Technologies Inc.
  • Keyence
  • Kyvos Insights
  • StreamSets
  • Striim
  • Syncsort
  • SK holdings C&C

    Contact us

    For conference registration information and customer service

    For more information on community discounts and trade opportunities with O’Reilly conferences

    For information on exhibiting or sponsoring a conference

    For media/analyst press inquires