Setting up a lightweight distributed caching layer using Apache Arrow

Description

Apache Arrow has quickly become the standard for high-performance in-memory processing. It has integration with major open source projects such as Spark, pandas, Parquet, Dremio, libgdf, and the GPU Open Analytics Initiative (GOAI). As the go-to representation for data processing and interchange, Arrow has substantially changed how well systems can share and process data. However, systems today only generate Arrow representation data ephemerally. The translation from on-disk formats to Arrow can diminish the overall performance potential.

Jacques Nadeau offers an overview of a new Apache-licensed lightweight distributed in-memory cache that allows multiple applications to consume Arrow directly using the Arrow RPC and IPC protocols. You’ll explore the system design and deployment architecture, including the cache lifecycle, update patterns, cache cohesion, and appropriate use cases; learn how data science, analytical, and custom applications can all leverage the cache simultaneously; discover the trade-offs around in-memory representations, data size, and balancing working memory with cache overhead; explore security, upgrades, and versioning, with a focus on how to balance performance, access, and governance; and see a live demo, showing the impact on overall performance and end-user satisfaction.