Schedule: Distributed Data & Databases sessions

The Netflix download feature allows users to download content for offline playback. Implementing this feature required a new persistence architecture to maintain the state of user devices and content licenses. Joseph Breuer and Robert Reta explore the technical decisions behind the choice of a Cassandra event sourcing data store. Read more.

Machine learning is as accessible as it has ever been, but it’s not always obvious how to go from a cool paper to serving production traffic. Ben Linsay helps you get started putting your paper into production, sharing lessons learned solving real problems with machine learning at Kickstarter. Read more.

Leif Walsh offers an overview of Flint, Two Sigma's open source time series extension to Spark, explains how it fits in with the Spark programming model, and lays out the roadmap for the future of pandas, PySpark, and Flint. Read more.

On the surface, adapting software to use persistent memory seems obvious. After all, persistent memory is simply fast memory that maintains state when the power goes out, like an SSD. But unlike SSDs, persistent memory challenges long-held ideas and conventions about how software works. Rob Dickinson outlines four key ideas that will help focus your persistent memory strategy. Read more.

Keystone, a critical piece of Netflix's backend data infrastructure, ensures massive data movements and real-time event processing. Zhenzhong Xu leads a deep dive into Keystone's architecture and underlying stream processing engines, sharing insights and proven paths on how the company achieves multitenancy, scalability, and resilience in a complex cloud-native distributed system environment. Read more.

Pinterest has to support real-time decision making while operating on petabyte-scale data. Swaminathan Sundaramurthy and Mark Cho offer an overview of Pinterest's real-time data pipeline (modeled on quasi-Kappa architecture), its impact on the company's systems, and tools and processes used and demonstrate how Pinterest models real-time ads analytics on the platform. Read more.

In the world of big and fast data, it's important to be fluent in storage and know the right tools for each job. Alex Petrov shares techniques for picking the right database and indexes, understanding the trade-offs different types of storage bring, scaling out your data and planning its growth, and finding the best resources on the subject. Read more.