Presented By
O’Reilly + Cloudera
Make Data Work
March 25-28, 2019
San Francisco, CA
Please log in

The magic behind your Lyft ride prices: A case study on machine learning and streaming

Rakesh Kumar (Lyft), Thomas Weise (Lyft)
5:10pm5:50pm Wednesday, March 27, 2019
Average rating: ****.
(4.00, 3 ratings)

Who is this presentation for?

  • Data scientists, ML engineers, data engineers, and technical decision makers



Prerequisite knowledge

  • A basic understanding of data processing and machine learning

What you'll learn

  • Learn about Lyft's machine learning-based dynamic pricing process, powered by a streaming platform


Lyft is a multidimensional marketplace. Not only does it require a buyer and a seller (a.k.a. a driver and a passenger); it also needs them to be interested at the same time and at the same place. Reacting to events with traditional methodologies is challenging, especially when timely reaction is required to balance the market condition. Thus, data science tools for machine learning and a process that allows for faster deployment is of growing importance to the business.

Rakesh Kumar and Thomas Weise discuss how Lyft uses its streaming platform to run dynamic pricing algorithms, allowing the company to be fair to drivers (by say, raising rates when there’s a lot of demand) and to passengers (by offering to return 10 minutes later for a cheaper rate). To accomplish this, the system consumes a massive amount of events from different sources.

Topics include:

  • Examples of dynamic pricing based on real-time event streams, including location of driver, ride requests, and user session events, and on machine learning models
  • A comparison of Lyft’s legacy system and its new streaming platform for dynamic pricing
  • A data scientist-friendly development environment using Python ecosystem tools that allows users to focus on the business logic
  • How to process live events in real time to generate features for machine learning models
  • An overview of the streaming platform architecture and technology stack
  • How to use Apache Beam portability framework as a bridge to distributed execution without a code rewrite for JVM-based streaming engine
  • Lessons learned
Photo of Rakesh Kumar

Rakesh Kumar


Rakesh Kumar is a software engineer on the pricing team at Lyft. He started his career as an embedded software engineer for mobile devices; later he moved to server-side engineer to tackle bigger challenges in distributed systems. His interests include machine learning and streaming systems.

Photo of Thomas Weise

Thomas Weise


Thomas Weise is a software engineer for the streaming platform at Lyft. He’s also a PMC member for the Apache Apex and Apache Beam projects and has contributed to several more projects within the ASF ecosystem. Thomas is a frequent speaker at international big data conferences and the author of Learning Apache Apex.