ETL and event sourcing

Description

Traditional ETL pipelines, consisting of extract, transform, and load stages, are a staple integration architecture pattern used in a wide variety of business domains. They present entire classes of familiar frustrations and impedance mismatches that many engineers have encountered firsthand.

More recently the concept of a data lake has grown in popularity, bringing certain ideas from domain-driven design, such as bounded contexts, to bear on these problem domains. Can you go even further into the world of DDD? What costs and benefits would you observe if you did?

Marc Siegel shares a real-life case study and lessons learned from going further into domain-driven design and applying the event sourcing pattern to the traditional problem domain of an ETL pipeline. If your work entails bringing lots of data into your system and building state off of it, you may find this talk interesting.

Topics include:

• ETL and its challenges
• Event sourcing basics
• Case study: Event sourcing in your ETL
• Lessons learned: Thinnest extractions possible
• Lessons learned: Extracted files as a source of truth
• Lessons learned: Better iteration on transformations
• Lessons learned: Why TL must be kept fast and run often