ETL 2.0: It’s not just for data engineers anymore

Robin Moffatt (Confluent)

15:00–15:45 Thursday, 7 November 2019

Location: M6/M7

Data

Secondary topics: Best Practice, Overview, Theoretical

Average rating:

(2.00, 2 ratings)

View slides

Who is this presentation for?

Architects, software engineers, and data engineers

Level

Beginner

Description

Developers have long employed message queues to decouple subsystems and provide an approximation of asynchronous processing. But these legacy queuing systems don’t adequately deliver on the promise of event-driven architectures and often lead to contrived integration patterns. Their limited scalability and durability limit you in using events to their full potential. Events carry both notification and state and form a powerful primitive upon which to build systems for developers and data engineers alike. Developers benefit from the asynchronous communication that events enable between services, and data engineers benefit from the integration capabilities. Everyone gains from using the standards-based, scalable, and resilient streaming platform.

Robin Moffatt explores the concepts of events, their relevance to software and data engineers, and their powers for unifying architectures in a powerful way. You’ll see how stream processing makes sense in both a microservices and ETL environment, and how ETL is actually more common than you may think. Robin provides a hands-on demonstration of these concepts in practice using Apache Kafka and commentary on the design choices made.

What you'll learn

Discover the power of events and unbounded data
Learn that streaming is not just for real-time applications—it’s for everyone, and you’re probably doing ETL without realizing it
Understand how event-driven architectures can enable greater scalability and flexibility of systems both now and future

Robin Moffatt

Confluent

Robin Moffatt is a senior developer advocate at Confluent, the company founded by the original creators of Apache Kafka, as well as an Oracle Groundbreaker Ambassador and ACE Director Alumnus. His career has always involved data, from the old worlds of COBOL and DB2 through the worlds of Oracle and Hadoop and into the current world with Kafka. His particular interests are analytics, systems architecture, performance testing, and optimization. He blogs at http://cnfl.io/rmoff and http://rmoff.net/ (and previously http://ritt.md/rmoff) and can be found tweeting grumpy geek thoughts as @rmoff. Outside of work, he enjoys drinking good beer and eating fried breakfasts, although generally not at the same time.