Herding cats: Product management in the machine learning era

Description

While the role of the manager doesn’t require deep knowledge of machine learning algorithms, it does require understanding how ML-based products should be developed and the ability to bridge the gap between product and business requirements and the inherent uncertainty at any ML-based solution. This uncertainty follows an ML-based solution at all phases: there’s rarely certainty that any ML-based solution can solve a given business problem.

The development cycle involves research iterations to continuously try improving the results—a cycle that isn’t deterministic. After the deployment of a solution, the results aren’t deterministic and may be erroneous, requiring handling of those in the product design and also careful monitoring of the ML performance (e.g., a model predicting churn of a subscriber outputs a probability for churn, which by itself may be wrong because of modeling issues, input data issues, etc.).

Ira Cohen walks you through the cycle of developing ML-based capabilities (or entire products) and the role of the (product) manager in each step of the cycle using real-world examples.

Outline:

Initial design

• Mapping a business problem to a ML solution (e.g., type of ML solution required, any constraints on learning time, runtime, and more)
• Mapping the outcomes of ML to the desired outcome from the user perspective (e.g., accuracy, visualization, and product flow wrapping the ML capability)

Research phase

• Data collection: Determining which data is available or should be produced
• Experimentation: Determining the success criteria
• How to know when to stop iterations

Deployment phase

• Requirements from the production process (real time versus offline, batch operations versus stream operations, etc.)
• How to roll out new versions and improvements

Production phase

• Monitoring requirements to ensure ML-based service performance
• Product design to handle uncertainty in outcomes of the ML component