The next data engineering architecture: Beyond the lake (sponsored by Thoughtworks)

Description

The data engineering domain has benefited from many advances over the recent years, including streaming technologies, on-demand infrastructure, and the proliferation of cloud-native tooling to digest, process, store, and serve data at scale. Despite these advances, one thing remains underdeveloped: the approach to the overall data engineering platform architecture and team structure. The current state of the art, unfortunately, remains centralized, functionally organized and managed by hyperspecialized teams, difficult to scale, and hard to leverage.

Zhamak Dehghani shares observations and lessons learned working with and consulting many organizations that are busy developing their data engineering capabilities and technology. Zhamak then introduces an alternative data engineering architecture, organizational, and operational model that can enable your organization to become truly data driven—a decentralized and ubiquitous data engineering platform that enables any team to produce or consume data at scale with low overhead.

This session is sponsored by Thoughtworks.