Real-time SQL stream processing at scale with Apache Kafka and KSQL
Who is this presentation for?
- Data engineers, developers, and database administrators
Level
Description
If you’ve ever thought you needed to be a programmer to do stream processing and build stream processing data pipelines, think again. Apache Kafka is a distributed, scalable, and fault-tolerant streaming platform, providing low-latency pub/sub messaging coupled with native storage and stream processing capabilities. Integrating Kafka with a relational database management system (RDBMS), NoSQL, and object stores is simple with Kafka Connect, which is part of Apache Kafka. KSQL is the open source SQL streaming engine for Apache Kafka and makes it possible to build stream processing applications at scale, written using a familiar SQL interface.
Viktor Gamov walks you through the architectural reasoning for Apache Kafka and the benefits of real-time integration. You’ll build a streaming data pipeline using nothing but your bare hands, Kafka Connect, and KSQL.
Prerequisite knowledge
- A basic understanding of SQL, databases, Linux and Shell, and Docker
Materials or downloads needed in advance
- A laptop
- Complete the "setup instructions":https://github.com/confluentinc/demo-scene/blob/master/ksql-workshop/pre-requisites.adoc
What you'll learn
- Discover best practices around building pipelines with Apache Kafka
- Learn how to use just config and SQL to build complete ETL pipelines
- Identify patterns for integration with databases and anti-patterns to be aware of
Viktor Gamov
Confluent
Viktor Gamov is a developer advocate at Confluent, the company that makes an event streaming platform based on Apache Kafka. Back in his consultancy days, Viktor developed comprehensive expertise in building enterprise application architectures using open source technologies. He enjoys helping architects and developers design and develop low-latency, scalable, and highly available distributed systems. He’s a professional conference speaker on distributed systems, streaming data, the Java virtual machine (JVM), and DevOps topics, and he’s a regular on events including JavaOne, Devoxx, OSCON, QCon, and others. He co-authored O’Reilly’s Enterprise Web Development. He blogs at gamov.io and cohosts Crazy Russians in Devoops and DevRelRad.io podcasts. Follow Viktor on Twitter as @gamussa, where he posts about gym life, food, open source, and, of course, Kafka and Confluent.
Comments on this page are now closed.
Presented by
Elite Sponsors
Strategic Sponsors
Zettabyte Sponsors
Contributing Sponsors
Exabyte Sponsors
Content Sponsor
Impact Sponsors
Supporting Sponsor
Non Profit
Contact us
confreg@oreilly.com
For conference registration information and customer service
partners@oreilly.com
For more information on community discounts and trade opportunities with O’Reilly conferences
strataconf@oreilly.com
For information on exhibiting or sponsoring a conference
pr@oreilly.com
For media/analyst press inquires
Comments
Hi Victor,
Where can I find the presentation slides of the talk?
Hi Victor,
Can you please share the
1) Console (name of the software) you have used for CLI
2) Presentation tool (extension I believe) for chrome where selecting text would give you options to Highlight selected text as Portrait and other userful options as a popup
worked for me as well
looks good after `git pull`
Hey Jayme
Could you update your local got copy with
got pull origin master
Or simply comment websocket and nginx containers on docker-compose.yml
No, still having the same issue. Is there a change I need to make on my end?
Hi Jayme,
The instructor said he was able to reproduce the issue and has fixed it.
Can you please try again, and if you’re still experiencing the same problem, let me know?
Thank you,
Sophia
I am having the exact same issue with the websockets pull.
I am having the exact same issue with the websockets pull.
Hello
I was trying to install the pre-requisites for this session.
Most things worked out.
but when trying to run `docker-compose pull` one of the modules is failing to download:
$ docker-compose pull
Pulling zookeeper … done
Pulling kafka … done
Pulling schema-registry … done
Pulling kafka-connect … done
Pulling ksql-server … done
Pulling ksql-cli … done
Pulling kafkacat … done
Pulling datagen-ratings … done
Pulling control-center … done
Pulling mysql … done
Pulling connect-debezium … done
Pulling elasticsearch … done
Pulling kibana … done
Pulling websockets … error
Pulling nginx … done
^T
ERROR: for websockets b"pull access denied for ksql-workshop-websockets, repository does not exist or may require ‘docker login’"
ERROR: pull access denied for ksql-workshop-websockets, repository does not exist or may require ‘docker login’