Running SQL-based workloads in the cloud at 20x–200x lower cost using Apache Arrow

Description

As companies move their workloads to cloud platforms like AWS and Azure, they must consider not only performance SLAs but also the requisite costs for delivering an SLA on a specific dataset. Likewise, as companies look to take advantage of cloud object stores like Amazon S3 and Azure ADLS, they must weigh the cost advantages of these services against the functional capabilities of legacy systems like the relational database.

In the EDW, indexes and materialized views are powerful features that allow the system to process queries orders-of-magnitude more efficiently than scanning full tables for each query. The power of these approaches lies in both their efficiency and their ability to be applied to different workloads without impacting the behavior of end users. Add an index or a materialized view, and most users will never need to change their query or the resources they connect to.

In the cloud, when storing data in systems like Amazon S3 and Azure ADLS, some SQL engines are available for querying the data, but these offerings don’t provide the same rich features in terms of indexes and materialized views.

Jacques Nadeau shares a novel solution that uses Apache Arrow, Apache Parquet, and Apache Calicite to provide features similar to materialized views in relational databases. The key differences are that this approach is integrated into the separation of compute and storage available on cloud platforms and scales to any number of nodes, works with nested data structures like JSON natively, and is fully open source.