Sep 9–12, 2019

Herding cats - product management in the Machine Learning era

Ira Cohen (Anodot)
9:00am12:30pm Tuesday, September 10, 2019
Location: 230 A

Who is this presentation for?

Product managers, Data Science managers, CTO, CIO




In the tutorial we will go through the cycle of developing machine learning based capabilities (or entire products) and the role of the (product) manager in each step of the cycle.

While the role of the manager does not require deep knowledge of machine learning algorithms – it does require understanding of how ML based products should be developed and bridge the gap between product/business requirements and the inherent uncertainty that is at the basis of any machine learning based solution. This uncertainty follows an ML based solution at all phases: there is rarely certainty that any ML based solution can solve a given business problem, the development cycle involves research iterations to continuously try improving the results – a cycle that is not deterministic. After deployment of a solution, the results are not deterministic and may be erroneous – requiring both handling of those in the product design but also careful monitoring of the ML performance (e.g., a model predicting churn of a subscriber outputs a probability for chrun, which by itself may be wrong because of modeling issues, input data issues, etc).

The development cycle of machine learning capabilities involves the following steps, with the product manager plays a crucial role in each of those:

1. Initial design:

a. Mapping a business problem to a ML solution (e.g., type of ML solution required, any constraints on learning time, runtime, and more )

Mapping the outcomes of ML to the desired outcome from the user perspective: e.g., accuracy, visualization, product flow wrapping the ML capability

2. Research phase:

a. Data collection: determining which data is available or should be produced

Experimentation: determining the success criteria

c. How to know when to stop iterations on i and ii

3. Deployment phase:

a. Requirements from the production process (real time vs offline, batch operations vs stream operations, etc).

How to rollout new versions and improvements.

4. Production phase:

a. Monitoring requirements to ensure the ML based service performance

Product design to handle the uncertainty in outcomes of the ML component.

We will go through these steps with examples from real products.

Prerequisite knowledge

Knowledge about the types of machine learning concepts and where they are applicable. For example, understanding the difference between supervised vs. unsupervised learning, types of ML solutions (classification, regression, clustering, etc).

Materials or downloads needed in advance

Laptop with internet connection.

What you'll learn

Three takeaways that every manager (PM/etc) should know: 1. How to determine when it is appropriate to use ML to solve a product/business problem? 2. How to map between the ML output and the needs of the product users? Especially bridging the gap between ML output and user expectations through product design. 3. Closing the loop: How to ensure that the ML product is functioning as intended for the users?
Photo of Ira Cohen

Ira Cohen


Ira Cohen is a cofounder and chief data scientist at Anodot, where he is responsible for developing and inventing the company’s real-time multivariate anomaly detection algorithms that work with millions of time series signals. He holds a PhD in machine learning from the University of Illinois at Urbana-Champaign and has over 12 years of industry experience.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)

Contact us

For conference registration information and customer service

For more information on community discounts and trade opportunities with O’Reilly conferences

Become a sponsor

For information on exhibiting or sponsoring a conference

Contact list

View a complete list of O'Reilly AI contacts