The magic behind your Lyft ride prices: A case study on machine learning and streaming

Description

Lyft is a multidimensional marketplace. Not only does it require a buyer and a seller (a.k.a. a driver and a passenger); it also needs them to be interested at the same time and at the same place. Reacting to events with traditional methodologies is challenging, especially when timely reaction is required to balance the market condition. Thus, data science tools for machine learning and a process that allows for faster deployment is of growing importance to the business.

Rakesh Kumar and Thomas Weise discuss how Lyft uses its streaming platform to run dynamic pricing algorithms, allowing the company to be fair to drivers (by say, raising rates when there’s a lot of demand) and to passengers (by offering to return 10 minutes later for a cheaper rate). To accomplish this, the system consumes a massive amount of events from different sources.

Topics include:

• Examples of dynamic pricing based on real-time event streams, including location of driver, ride requests, and user session events, and on machine learning models
• A comparison of Lyft’s legacy system and its new streaming platform for dynamic pricing
• A data scientist-friendly development environment using Python ecosystem tools that allows users to focus on the business logic
• How to process live events in real time to generate features for machine learning models
• An overview of the streaming platform architecture and technology stack
• How to use Apache Beam portability framework as a bridge to distributed execution without a code rewrite for JVM-based streaming engine
• Lessons learned