Sep 23–26, 2019

Real-time SQL stream processing at scale with Apache Kafka and KSQL

Viktor Gamov (Confluent)
9:00am12:30pm Tuesday, September 24, 2019
Location: 1E 10
Average rating: ***..
(3.50, 6 ratings)

Who is this presentation for?

  • Data engineers, developers, and database administrators

Level

Intermediate

Description

If you’ve ever thought you needed to be a programmer to do stream processing and build stream processing data pipelines, think again. Apache Kafka is a distributed, scalable, and fault-tolerant streaming platform, providing low-latency pub/sub messaging coupled with native storage and stream processing capabilities. Integrating Kafka with a relational database management system (RDBMS), NoSQL, and object stores is simple with Kafka Connect, which is part of Apache Kafka. KSQL is the open source SQL streaming engine for Apache Kafka and makes it possible to build stream processing applications at scale, written using a familiar SQL interface.

Viktor Gamov walks you through the architectural reasoning for Apache Kafka and the benefits of real-time integration. You’ll build a streaming data pipeline using nothing but your bare hands, Kafka Connect, and KSQL.

Prerequisite knowledge

  • A basic understanding of SQL, databases, Linux and Shell, and Docker

Materials or downloads needed in advance

  • A laptop
  • Complete the "setup instructions":https://github.com/confluentinc/demo-scene/blob/master/ksql-workshop/pre-requisites.adoc

What you'll learn

  • Discover best practices around building pipelines with Apache Kafka
  • Learn how to use just config and SQL to build complete ETL pipelines
  • Identify patterns for integration with databases and anti-patterns to be aware of
Photo of Viktor Gamov

Viktor Gamov

Confluent

Viktor Gamov is a developer advocate at Confluent, the company that makes an event streaming platform based on Apache Kafka. Back in his consultancy days, Viktor developed comprehensive expertise in building enterprise application architectures using open source technologies. He enjoys helping architects and developers design and develop low-latency, scalable, and highly available distributed systems. He’s a professional conference speaker on distributed systems, streaming data, the Java virtual machine (JVM), and DevOps topics, and he’s a regular on events including JavaOne, Devoxx, OSCON, QCon, and others. He co-authored O’Reilly’s Enterprise Web Development. He blogs at gamov.io and cohosts Crazy Russians in Devoops and DevRelRad.io podcasts. Follow Viktor on Twitter as @gamussa, where he posts about gym life, food, open source, and, of course, Kafka and Confluent.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)

Comments

miguel farrajota |
10/01/2019 4:42am EDT

Hi Victor,

Where can I find the presentation slides of the talk?

Srujan gulla | Senior Software Engineer
09/27/2019 7:06am EDT

Hi Victor,

Can you please share the
1) Console (name of the software) you have used for CLI
2) Presentation tool (extension I believe) for chrome where selecting text would give you options to Highlight selected text as Portrait and other userful options as a popup

Jayme McBride | Senior Director Data Engineering
09/24/2019 4:35am EDT

worked for me as well

Dmitry Belyavsky | Sr. Developer/Tech Lead
09/23/2019 6:57pm EDT

looks good after `git pull`

Picture of Viktor Gamov
Viktor Gamov | Senior Solutions Architect
09/23/2019 6:58am EDT

Hey Jayme

Could you update your local got copy with

got pull origin master

Or simply comment websocket and nginx containers on docker-compose.yml

Jayme McBride | Senior Director Data Engineering
09/23/2019 6:17am EDT

No, still having the same issue. Is there a change I need to make on my end?

Picture of Sophia DeMartini
Sophia DeMartini | Senior Speaker Manager
09/20/2019 1:41pm EDT

Hi Jayme,

The instructor said he was able to reproduce the issue and has fixed it.

Can you please try again, and if you’re still experiencing the same problem, let me know?

Thank you,
Sophia

Jayme McBride | Sr. Director
09/20/2019 12:31pm EDT

I am having the exact same issue with the websockets pull.

Jayme McBride | Sr. Director
09/20/2019 12:30pm EDT

I am having the exact same issue with the websockets pull.

Dmitry Belyavsky | Sr. Developer/Tech Lead
09/17/2019 7:21pm EDT

Hello

I was trying to install the pre-requisites for this session.
Most things worked out.
but when trying to run `docker-compose pull` one of the modules is failing to download:

$ docker-compose pull
Pulling zookeeper … done
Pulling kafka … done
Pulling schema-registry … done
Pulling kafka-connect … done
Pulling ksql-server … done
Pulling ksql-cli … done
Pulling kafkacat … done
Pulling datagen-ratings … done
Pulling control-center … done
Pulling mysql … done
Pulling connect-debezium … done
Pulling elasticsearch … done
Pulling kibana … done
Pulling websockets … error
Pulling nginx … done
^T
ERROR: for websockets b"pull access denied for ksql-workshop-websockets, repository does not exist or may require ‘docker login’"
ERROR: pull access denied for ksql-workshop-websockets, repository does not exist or may require ‘docker login’

    Contact us

    confreg@oreilly.com

    For conference registration information and customer service

    partners@oreilly.com

    For more information on community discounts and trade opportunities with O’Reilly conferences

    strataconf@oreilly.com

    For information on exhibiting or sponsoring a conference

    pr@oreilly.com

    For media/analyst press inquires