Presented By
O’Reilly + Cloudera
Make Data Work
29 April–2 May 2019
London, UK
Please log in

Streaming at Lyft

Thomas Weise (Lyft)
11:1511:55 Thursday, 2 May 2019
Data Engineering and Architecture, Expo Hall, Streaming and IoT
Location: Expo Hall 2 (Capital Hall N24)
Average rating: ****.
(4.50, 14 ratings)

Who is this presentation for?

  • Data engineers, architects, and technical decision makers



Prerequisite knowledge

  • A basic understanding of data infrastructure and stream processing systems

What you'll learn

  • Learn about Lyft’s streaming platform, its key components, and how the platform is used to power Lyft rides


Lyft’s systems need to track and react to event streams in real time to update locations, compute routes and estimates, balance prices, and more. Reacting to events with traditional methodologies is a challenge, especially in cases where low-latency SLAs for instant user feedback are important. As a result, fast data and stream processing are essential for making Lyft rides a good experience for passengers and drivers.

Thomas Weise offers an overview of the streaming platform that powers these use cases.

Topics include:

  • A deep dive into Lyft’s streaming platform, with use cases, system architecture, and key requirements that drive technology choices
  • Streaming source and event storage with Apache Kafka and S3—why both are needed for replay, backfill, and bootstrapping
  • Stateful streaming computation with scalability, high availability, and low-latency processing on Apache Flink
  • Streaming application development frameworks, abstraction levels, and the language to use case fit for Java, SQL, and Python
  • Python on Apache Beam as the bridge from a data science and machine learning friendly environment to distributed execution on Flink
  • Kubernetes-based deployment to abstract infrastructure and simplify operations of stateful Flink applications
Photo of Thomas Weise

Thomas Weise


Thomas Weise is a software engineer for the streaming platform at Lyft. He’s also a PMC member for the Apache Apex and Apache Beam projects and has contributed to several more projects within the ASF ecosystem. Thomas is a frequent speaker at international big data conferences and the author of Learning Apache Apex.