Apache Arrow has quickly become the standard for high-performance in-memory processing. It has integration with major open source projects such as Spark, pandas, Parquet, Dremio, libgdf, and the GPU Open Analytics Initiative (GOAI). As the go-to representation for data processing and interchange, Arrow has substantially changed how well systems can share and process data. However, systems today only generate Arrow representation data ephemerally. The translation from on-disk formats to Arrow can diminish the overall performance potential.
Jacques Nadeau offers an overview of a new Apache-licensed lightweight distributed in-memory cache that allows multiple applications to consume Arrow directly using the Arrow RPC and IPC protocols. You’ll explore the system design and deployment architecture, including the cache lifecycle, update patterns, cache cohesion, and appropriate use cases; learn how data science, analytical, and custom applications can all leverage the cache simultaneously; discover the trade-offs around in-memory representations, data size, and balancing working memory with cache overhead; explore security, upgrades, and versioning, with a focus on how to balance performance, access, and governance; and see a live demo, showing the impact on overall performance and end-user satisfaction.
Jacques Nadeau is the cofounder and CTO of Dremio. Previously, he ran MapR’s distributed systems team; was CTO and cofounder of YapMap, an enterprise search startup; and held engineering leadership roles at Quigo, Offermatica, and aQuantive. Jacques is cocreator and PMC chair of Apache Arrow, a PMC member of Apache Calcite, a mentor for Apache Heron, and the founding PMC chair of the open source Apache Drill project.
©2018, O’Reilly UK Ltd • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com