Presented By O'Reilly and Cloudera
Make Data Work
March 13–14, 2017: Training
March 14–16, 2017: Tutorials & Conference
San Jose, CA
Network
Join Attendee Network
 

Netflix Keystone SPaaS: Real-time stream processing as a service

Monal Daxini (Netflix)
5:10pm–5:50pm Wednesday, March 15, 2017
Data engineering and architecture
 Location: LL20 A Level: Advanced
Secondary topics:  Architecture, Media, Platform, Streaming
Average rating: ****.
(4.50, 2 ratings)

Who is this presentation for?

  • Big data engineers, senior analytics engineers, and senior software engineers

Prerequisite knowledge

  • Familiarity building any data ingest pipeline
  • Basic knowledge of Kafka and the stream processing domain (useful but not required)

What you'll learn

  • Explore how Netflix is building a scalable, self-service, fault-tolerant multitenant stream processing service in the cloud by leveraging Kafka, Samza, Flink, Mesos, and Docker and how it is evolving the service in flight without downtime

Description

The need for gleaning answers from data in real time from unbounded data streams is moving from nicety to a necessity. Netflix is a data-driven company, and Netflix Keystone SPaaS processes over a trillion messages a day with at-least-once processing semantics in the cloud to derive insights. To ease extracting intelligence from this unbounded stream easily, Netflix is building stream-processing-as-a-service (SPaaS) infrastructure so that the user can focus on extracting value and not have to worry about building a scalable, fault-tolerant infrastructure.

Monal Daxini explores how Netflix is building a scalable, self-serve, fault-tolerant, multitenant stream processing service by leveraging open source software like Kafka, Samza, Flink, Mesos, and Docker, how it is evolving the service in flight without downtime, and lessons learned along the way.

Photo of Monal Daxini

Monal Daxini

Netflix

Monal Daxini is an engineering manager at Netflix, where he is building a scalable and multitenant event processing pipeline and leads the infrastructure for stream processing as a service. He has worked on Netflix’s Cassandra and Dynamite infrastructure and was instrumental in developing the encoding compute infrastructure for all Netflix content. Monal has 15 years of experience building distributed systems at organizations like Netflix, NFL.com, and Cisco.