Herding cats - product management in the Machine Learning era
Who is this presentation for?Product managers, Data Science managers, CTO, CIO
In the tutorial we will go through the cycle of developing machine learning based capabilities (or entire products) and the role of the (product) manager in each step of the cycle.
While the role of the manager does not require deep knowledge of machine learning algorithms – it does require understanding of how ML based products should be developed and bridge the gap between product/business requirements and the inherent uncertainty that is at the basis of any machine learning based solution. This uncertainty follows an ML based solution at all phases: there is rarely certainty that any ML based solution can solve a given business problem, the development cycle involves research iterations to continuously try improving the results – a cycle that is not deterministic. After deployment of a solution, the results are not deterministic and may be erroneous – requiring both handling of those in the product design but also careful monitoring of the ML performance (e.g., a model predicting churn of a subscriber outputs a probability for chrun, which by itself may be wrong because of modeling issues, input data issues, etc).
The development cycle of machine learning capabilities involves the following steps, with the product manager plays a crucial role in each of those:
1. Initial design:
a. Mapping a business problem to a ML solution (e.g., type of ML solution required, any constraints on learning time, runtime, and more )Mapping the outcomes of ML to the desired outcome from the user perspective: e.g., accuracy, visualization, product flow wrapping the ML capability
2. Research phase:
a. Data collection: determining which data is available or should be producedExperimentation: determining the success criteria
c. How to know when to stop iterations on i and ii
3. Deployment phase:
a. Requirements from the production process (real time vs offline, batch operations vs stream operations, etc).How to rollout new versions and improvements.
4. Production phase:
a. Monitoring requirements to ensure the ML based service performanceProduct design to handle the uncertainty in outcomes of the ML component.
We will go through these steps with examples from real products.
Prerequisite knowledgeKnowledge about the types of machine learning concepts and where they are applicable. For example, understanding the difference between supervised vs. unsupervised learning, types of ML solutions (classification, regression, clustering, etc).
Materials or downloads needed in advance
What you'll learnThree takeaways that every manager (PM/etc) should know: 1. How to determine when it is appropriate to use ML to solve a product/business problem? 2. How to map between the ML output and the needs of the product users? Especially bridging the gap between ML output and user expectations through product design. 3. Closing the loop: How to ensure that the ML product is functioning as intended for the users?
Ira Cohen is a cofounder and chief data scientist at Anodot, where he is responsible for developing and inventing the company’s real-time multivariate anomaly detection algorithms that work with millions of time series signals. He holds a PhD in machine learning from the University of Illinois at Urbana-Champaign and has over 12 years of industry experience.
Leave a Comment or Question
Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?
Join the conversation here (requires login)
For conference registration information and customer service
For more information on community discounts and trade opportunities with O’Reilly conferences
For information on exhibiting or sponsoring a conference
View a complete list of O'Reilly AI contacts