Data reflections: Making data fast and easy to use without making copies

Description

Raw data is rarely suitable for business users or data scientists. First, it is typically spread across many systems and may not be be accessible to those who want to consume the data. Second, it is often structured to suit the needs of the application developer, as opposed to the data consumer. Finally, it is usually not readily available in a single system that can magically respond to any analytical query at interactive speed.

In order to make data available, companies develop complex ETL pipelines in which data is copied many times between systems. For example, in order to achieve high performance, a subset of the overall dataset may be copied into a relational data warehouse, and then preaggregated into aggregation tables or external cubes or extracted into BI servers. This leads to tremendous cost and impacts the organization’s agility. It also eliminates the possibility of self-service, because it becomes impossible for a data consumer to know which system and table they should utilize to answer a specific question. As a result, in most enterprises, it can take months to develop a new BI dashboard or machine learning model.

Tomer Shiran and Jacques Nadeau introduce a new approach called data reflections, which dramatically reduces the need for data copies. When used in conjunction with a cost-based optimizer such as Apache Calcite, data reflections can help accelerate queries without the need for data engineers to manually create data copies or data consumers to interact with different materializations of data to achieve the desired performance. In addition, data reflections provide separation between the logical world, where analysts and data scientists need to curate and transform the model of the data, and and the physical world, where data must be physically optimized in order to enable execution engines to respond to queries in real-time.

In addition to providing an overview of data reflections and explaining the technological underpinnings, Tomer and Jacques offer a live demo of an open source implementation that shows how data science workloads, such as those in Python/pandas and R, and BI workloads can be automatically accelerated without having to explicitly move or copy data and while affording users the freedom to curate and transform the logical model of the data.