4–7 Nov 2019

ETL 2.0: It’s not just for data engineers anymore

Robin Moffatt (Confluent)
15:0015:45 Thursday, 7 November 2019
Location: M6/M7
Secondary topics:  Best Practice, Overview, Theoretical

Who is this presentation for?

  • Architects, software engineers, and data engineers

Level

Beginner

Description

Developers have long employed message queues to decouple subsystems and provide an approximation of asynchronous processing. But these legacy queuing systems don’t adequately deliver on the promise of event-driven architectures and often lead to contrived integration patterns. Their limited scalability and durability limit you in using events to their full potential. Events carry both notification and state and form a powerful primitive upon which to build systems for developers and data engineers alike. Developers benefit from the asynchronous communication that events enable between services, and data engineers benefit from the integration capabilities. Everyone gains from using the standards-based, scalable, and resilient streaming platform.

Robin Moffatt explores the concepts of events, their relevance to software and data engineers, and their powers for unifying architectures in a powerful way. You’ll see how stream processing makes sense in both a microservices and ETL environment, and how ETL is actually more common than you may think. Robin provides a hands-on demonstration of these concepts in practice using Apache Kafka and commentary on the design choices made.

What you'll learn

  • Discover the power of events and unbounded data
  • Learn that streaming is not just for real-time applications—it’s for everyone, and you’re probably doing ETL without realizing it
  • Understand how event-driven architectures can enable greater scalability and flexibility of systems both now and future
Photo of Robin Moffatt

Robin Moffatt

Confluent

Robin Moffatt is a developer advocate at Confluent, the company founded by the original creators of Apache Kafka, as well as an Oracle Groundbreaker Ambassador and ACE Director Alumnus. His career has always involved data, from the old worlds of COBOL and DB2 through the worlds of Oracle and Hadoop and into the current world with Kafka. His particular interests are analytics, systems architecture, performance testing, and optimization. He blogs at http://cnfl.io/rmoff and http://rmoff.net/ (and previously http://ritt.md/rmoff) and can be found tweeting grumpy geek thoughts as @rmoff. Outside of work, he enjoys drinking good beer and eating fried breakfasts, although generally not at the same time.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)

Contact us

confreg@oreilly.com

For conference registration information and customer service

partners@oreilly.com

For more information on community discounts and trade opportunities with O’Reilly conferences

Become a sponsor

For information on exhibiting or sponsoring a conference

pr@oreilly.com

For media/analyst press inquires