Traditional ETL pipelines, consisting of extract, transform, and load stages, are a staple integration architecture pattern used in a wide variety of business domains. They present entire classes of familiar frustrations and impedance mismatches that many engineers have encountered firsthand.
More recently the concept of a data lake has grown in popularity, bringing certain ideas from domain-driven design, such as bounded contexts, to bear on these problem domains. Can you go even further into the world of DDD? What costs and benefits would you observe if you did?
Marc Siegel shares a real-life case study and lessons learned from going further into domain-driven design and applying the event sourcing pattern to the traditional problem domain of an ETL pipeline. If your work entails bringing lots of data into your system and building state off of it, you may find this talk interesting.
Marc Siegel is an engineering manager at Panorama Education, an education technology firm based in Boston. He has experience in web applications and event-driven systems, sometimes simultaneously. He’s passionate about building systems that tell the truth when asked questions. He got his start developing heterogeneous mobile network nodes at MIT Lincoln Laboratory and led development efforts at a number of startups, for everything from bidding systems for internet advertising to mobile handheld inspection software for tower cranes, although he is afraid of heights. Marc holds a BS in computer science from Brown University. You can find the slides from his last talk at the O’Reilly Software Architecture conference here.
Comments on this page are now closed.
©2019, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org