Real-time SQL stream processing at scale with Apache Kafka and KSQL
Who is this presentation for?
- Data engineers, developers, and database administrators
If you’ve ever thought you needed to be a programmer to do stream processing and build stream processing data pipelines, think again. Apache Kafka is a distributed, scalable, and fault-tolerant streaming platform, providing low-latency pub/sub messaging coupled with native storage and stream processing capabilities. Integrating Kafka with a relational database management system (RDBMS), NoSQL, and object stores is simple with Kafka Connect, which is part of Apache Kafka. KSQL is the open source SQL streaming engine for Apache Kafka and makes it possible to build stream processing applications at scale, written using a familiar SQL interface.
Viktor Gamov walks you through the architectural reasoning for Apache Kafka and the benefits of real-time integration. You’ll build a streaming data pipeline using nothing but your bare hands, Kafka Connect, and KSQL.
- A basic understanding of SQL, databases, Linux and Shell, and Docker
Materials or downloads needed in advance
- A laptop
- Complete the "setup instructions":https://github.com/confluentinc/demo-scene/blob/master/ksql-workshop/pre-requisites.adoc
What you'll learn
- Discover best practices around building pipelines with Apache Kafka
- Learn how to use just config and SQL to build complete ETL pipelines
- Identify patterns for integration with databases and anti-patterns to be aware of
Viktor Gamov is a developer advocate at Confluent, the company that makes an event streaming platform based on Apache Kafka. Back in his consultancy days, Viktor developed comprehensive expertise in building enterprise application architectures using open source technologies. He enjoys helping architects and developers design and develop low-latency, scalable, and highly available distributed systems. He’s a professional conference speaker on distributed systems, streaming data, the Java virtual machine (JVM), and DevOps topics, and he’s a regular on events including JavaOne, Devoxx, OSCON, QCon, and others. He co-authored O’Reilly’s Enterprise Web Development. He blogs at gamov.io and cohosts Crazy Russians in Devoops and DevRelRad.io podcasts. Follow Viktor on Twitter as @gamussa, where he posts about gym life, food, open source, and, of course, Kafka and Confluent.
Leave a Comment or Question
Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?
Join the conversation here (requires login)
For conference registration information and customer service
For more information on community discounts and trade opportunities with O’Reilly conferences
For information on exhibiting or sponsoring a conference
For media/analyst press inquires