Presented By O’Reilly and Cloudera
Make Data Work
21–22 May 2018: Training
22–24 May 2018: Tutorials & Conference
London, UK

Executive Briefing: Why machine-learned models crash and burn in production and what to do about it

David Talby (Pacific AI)
17:2518:05 Wednesday, 23 May 2018
Data-driven business management, Executive Briefing, Strata Business Summit
Location: Capital Suite 17 Level: Intermediate
Secondary topics:  Managing and Deploying Machine Learning
Average rating: ****.
(4.00, 1 rating)

Who is this presentation for?

  • Executives, chief architects, product and engineering leaders, and anyone building data science-based teams and products

Prerequisite knowledge

  • Familiarity with machine learning at a business/product level

What you'll learn

  • Understand the key risks and best practices to consider when building machine learning-intensive products and teams


Much progress has been made over the past decade on process and tooling for managing large-scale, multitier cloud apps and APIs, but there is far less common knowledge on best practices for managing machine-learned models (classifiers, forecasters, etc.), especially beyond the modeling, optimization, and deployment process once these models are in production.

A key mindset shift required to address these issues is understanding that model development is different than software development in fundamental ways. David Talby shares real-world case studies demonstrating best practices that executives, solution architects, and delivery teams must take into account when committing to successfully deliver and operate data science intensive systems in the real world and discusses lessons learned from a decade of experience building and operating such systems at Fortune 500 companies across several industries.

Topics include:

  • Concept drift: Adapting models to a changing environment
  • Locality and limited reuse and generalization of models
  • A/B testing challenges, which make it very hard in practice to know which model will perform better in production
  • Semisupervised and adversarial learning scenarios, which require modeling and optimizing models only once they’re in production
  • The impact of all of the above on product planning, staffing, and client expectation management
Photo of David Talby

David Talby

Pacific AI

David Talby is a chief technology officer at Pacific AI, helping fast-growing companies apply big data and data science techniques to solve real-world problems in healthcare, life science, and related fields. David has extensive experience in building and operating web-scale data science and business platforms, as well as building world-class, agile, distributed teams. Previously, he led business operations for Bing Shopping in the US and Europe with Microsoft’s Bing Group and built and ran distributed teams that helped scale Amazon’s financial systems with Amazon in both Seattle and the UK. David holds a PhD in computer science and master’s degrees in both computer science and business administration.