The changing face of ETL: Event-driven architectures for data engineers

Description

Data integration in architectures built on static, update-in-place data stores inevitably end up with pathologically high degrees of coupling and poor scalability. This has been the standard practice for decades as we attempt to build data pipelines on top of databases that do a poor job modeling the fundamental objects that drive our businesses and systems: events.

Events carry both notification and state and form a powerful primitive on which to build systems for developers and data engineers alike. Developers benefit from the asynchronous communication that events enable between services, and data engineers benefit from the integration capabilities. Everyone gains from using the scalable and resilient standards-based streaming platform.

Robin Moffatt discusses the concepts of events, their relevance to software and data engineers, and their ability to unify architectures in a powerful way. Join in to learn why analytics, data integration, and ETL fit naturally into a streaming world. Along the way, Robin will lead a hands-on demonstration of these concepts in practice and commentary on the design choices made.

Topics include:

• The power of events and unbounded data
• Streaming is not just for real-time applications; it’s for everyone
• Where a streaming platform fits in an analytic architecture
• How event-driven architectures can enable greater scalability and flexibility of systems both now and in the future