Presented By
O’Reilly + Cloudera
Make Data Work
March 25-28, 2019
San Francisco, CA

Professional Kafka development (Day 2)

Jesse Anderson (Big Data Institute)
Location: 3016

Who is this presentation for?

Software Engineers, Data Engineers, Data Scientists



Prerequisite knowledge

Programming (Java) A knowledge of other Big Data technologies is helpful but not required

What you'll learn

How to create large scale real-time systems using Apache Kafka How real-time distributed systems are different from batch systems How to create Kafka producers and consumers How to use Apache Avro with Kafka for truly enterprise grade solutions How to use Kafka Streams to create data pipelines How to configure Kafka Connect to put data into and take out of Kafka How to use your existing SQL skills with KSQL The best practices and common architectural patterns when creating solutions with Kafka


Day 1

Data At Scale

  • Data Movement Concepts
  • Moving Data at Scale

Kafka Concepts

  • Kafka System
  • Basic Concepts
  • Advanced Concepts

Developing With Kafka

  • Using Apache Maven
  • Kafka API
  • Kafka API Caveats

Advanced Kafka Development

  • Advanced Consumers and Producers
  • Advanced Offset Handling
  • Transactions
  • Multi-threading Consumers

Day 2

Kafka and Avro

  • Why Serialize
  • Avro and Serialization Formats

Kafka Connect

  • Using Kafka Connect
  • Importing From JDBC
  • Exporting to HDFS

Kafka Streams

  • Kafka Streams
  • Kafka Streams API


  • Using KSQL


Photo of Jesse Anderson

Jesse Anderson

Big Data Institute

Jesse Anderson is a data engineer, creative engineer, and managing director of the Big Data Institute. Jesse trains employees on big data—including cutting-edge technology like Apache Kafka, Apache Hadoop, and Apache Spark. He’s taught thousands of students at companies ranging from startups to Fortune 100 companies the skills to become data engineers. He’s widely regarded as an expert in the field and recognized for his novel teaching practices. Jesse is published by O’Reilly and Pragmatic Programmers and has been covered in such prestigious media outlets as the Wall Street Journal, CNN, BBC, NPR, Engadget, and Wired. You can learn more about Jesse at