Schedule: Real time, events, streams & scale sessions

Rob Claire explores the the technical challenges and lessons learned in building a monitoring stack that can reliably process millions of events per second, covering specific technologies—including Spark Streaming, Kafka, and HBase—and best practices for managing and monitoring data. Read more.

Keystone, a critical piece of Netflix's backend data infrastructure, ensures massive data movements and real-time event processing. Zhenzhong Xu leads a deep dive into Keystone's architecture and underlying stream processing engines, sharing insights and proven paths on how the company achieves multitenancy, scalability, and resilience in a complex cloud-native distributed system environment. Read more.

Pinterest has to support real-time decision making while operating on petabyte-scale data. Swaminathan Sundaramurthy and Mark Cho offer an overview of Pinterest's real-time data pipeline (modeled on quasi-Kappa architecture), its impact on the company's systems, and tools and processes used and demonstrate how Pinterest models real-time ads analytics on the platform. Read more.

Twitter is powered by thousands of microservices running on an internal cloud platform, which offer compute, storage, messaging, monitoring, etc. as a service. Vinu Charanya explains how she and her team are building a system that captures, defines, provisions, meters, and charges infrastructure resources, redefining how systems are built atop Twitter infrastructure. Read more.