Presented By O’Reilly and Cloudera
Make Data Work
21–22 May 2018: Training
22–24 May 2018: Tutorials & Conference
London, UK

Setting up a lightweight distributed caching layer using Apache Arrow

Jacques Nadeau (Dremio)
12:0512:45 Thursday, 24 May 2018
Average rating: ****.
(4.00, 3 ratings)

Who is this presentation for?

  • Data consumers, data scientists, data engineers, and developers

What you'll learn

  • Learn how Apache Arrow can speed access to data for multiple purposes


Apache Arrow has quickly become the standard for high-performance in-memory processing. It has integration with major open source projects such as Spark, pandas, Parquet, Dremio, libgdf, and the GPU Open Analytics Initiative (GOAI). As the go-to representation for data processing and interchange, Arrow has substantially changed how well systems can share and process data. However, systems today only generate Arrow representation data ephemerally. The translation from on-disk formats to Arrow can diminish the overall performance potential.

Jacques Nadeau offers an overview of a new Apache-licensed lightweight distributed in-memory cache that allows multiple applications to consume Arrow directly using the Arrow RPC and IPC protocols. You’ll explore the system design and deployment architecture, including the cache lifecycle, update patterns, cache cohesion, and appropriate use cases; learn how data science, analytical, and custom applications can all leverage the cache simultaneously; discover the trade-offs around in-memory representations, data size, and balancing working memory with cache overhead; explore security, upgrades, and versioning, with a focus on how to balance performance, access, and governance; and see a live demo, showing the impact on overall performance and end-user satisfaction.

Photo of Jacques Nadeau

Jacques Nadeau


Jacques Nadeau is the cofounder and CTO of Dremio. Previously, he ran MapR’s distributed systems team; was CTO and cofounder of YapMap, an enterprise search startup; and held engineering leadership roles at Quigo, Offermatica, and aQuantive. Jacques is cocreator and PMC chair of Apache Arrow, a PMC member of Apache Calcite, a mentor for Apache Heron, and the founding PMC chair of the open source Apache Drill project.