Build & maintain complex distributed systems
October 1–2, 2017: Training
October 2–4, 2017: Tutorials & Conference
New York, NY

Running a Massively Parallel Stream Processing System at Netflix

Zhenzhong Xu (Netflix)
1:30pm2:10pm Wednesday, October 4, 2017
Distributed Data & Databases, Real time, events, streams & scale
Location: Grand Ballroom West Level: Advanced

Who is this presentation for?

Software Engineers, Architects, data engineers, data infrastructure

Prerequisite knowledge

Basic understanding of distributed system, streaming processing concepts. Basic understanding of cloud-native microservices architectures. Working knowledge of Apache Kafka (or other similar replayable streaming source).

What you'll learn

This presentation will educate developers and architects how Netflix approaches big data streaming infrastructure, from high-level architecture to operations and toolings in the ecosystem.

Description

Over 200 million devices worldwide are capable of streaming Netflix content. Sitting on top of a microservice architecture, the entire ecosystem generates 1+ trillion events every day to feed critical Netflix systems to monitor service health, to detect fraudulent behaviors, and to improve customer experience, etc.

Keystone is the critical piece of backend infrastructure to ensure massive amount of events are delivered in near real time reliably, at scale, and in face of failures.

In this talk, we will deep dive into Keystone’s architecture and underlying stream processing engines. The talk will provide insights and proven paths on how we achieved multi-tenancy, scalability and resilience in a cloud native complex distributed system environment, and learnings we had.

Photo of Zhenzhong Xu

Zhenzhong Xu

Netflix

Zhenzhong Xu is currently a Software Engineer working on highly scalable and resilient streaming data infrastructure at Netflix. Previously, he was a core contributor to Microsoft Azure datacenter operating system reconciliation management & resiliency functionalities. He is passionate about anything related to real time data systems & large scale distributed system.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)