Setting up a lightweight distributed caching layer using Apache Arrow

Description

Apache Arrow has quickly become the standard for high-performance in-memory processing. It has integration into major open source projects such as Spark, pandas, Parquet, Dremio, libgdf, and the GPU Open Analytics Initiative (GoAi). All of these projects have adopted Arrow as the go-to representation for data processing and interchange, which has substantially changed how well systems can share and process data. However, systems today only generate Arrow representation data ephemerally. The translation from on-disk formats to Arrow can diminish the overall performance potential.

Jacques Nadeau leads a deep dive into a new Apache-licensed lightweight distributed in-memory cache that allows multiple applications to consume Arrow directly using the Arrow RPC and IPC protocols. Jacques offers an overview of the system design and deployment architecture, focusing on the cache lifecycle, update patterns, cache cohesion, and appropriate use cases. He then explores in detail how data science, analytical, and custom applications can all leverage the cache simultaneously as well as the trade-offs around in-memory representations, data size, and balancing working memory with cache overhead. Along the way, Jacques discusses security, upgrades, and versioning, focusing on how to balance performance, access, and governance. Jacques concludes with a demonstration of the cache in action, showing its impact on overall performance and ultimately, end-user satisfaction.