Mar 15–18, 2020

Model governance: A checklist for getting AI safely to production

David Talby (Pacific AI)
4:15pm4:55pm Tuesday, March 17, 2020
Location: LL21 E/F

Who is this presentation for?

Data scientists or analysts




The software industry has gone through multiple generations of tools and methodologies for software code governance: configuration control, collaboration, test processes, build processes, code repositories, and metadata management. In contrast, we’re just starting to explore these issues for machine learning models—which is already hurting the productivity of data science teams and preventing the safe deployment and operation of models in production.

David Talby summarizes current best practices for model governance, along with freely available tools you can use today to apply them. You’ll explore storing modeling assets in a searchable catalog, including notebooks, datasets, resulting measurements, hyperparameters, and other metadata; enabling reproducibility and sharing of experiments across data science team members; versioning models advanced beyond an experiment to a release candidate; testing models that are candidates for production use for accuracy, bias, and stability; validating models before launching in new geographies or populations; building and versioning the inference services of a model, including all underlying libraries and dependencies, as part of a standard CI/CD pipeline; the ability to release a model—roll out, roll back, or have multiple live versions; security, role-based access control, and an approval workflow for model release; storing and providing all metadata needed for a full audit trail.

Prerequisite knowledge

  • A basic understanding of the machine learning and deep learning model development lifecycle

What you'll learn

  • Discover applicable best practices and tools for managing a data science team and deploying and operating models into production
Photo of David Talby

David Talby

Pacific AI

David Talby is a chief technology officer at Pacific AI, helping fast-growing companies apply big data and data science techniques to solve real-world problems in healthcare, life science, and related fields. David has extensive experience in building and operating web-scale data science and business platforms, as well as building world-class, agile, distributed teams. Previously, he led business operations for Bing Shopping in the US and Europe with Microsoft’s Bing Group and built and ran distributed teams that helped scale Amazon’s financial systems with Amazon in both Seattle and the UK. David holds a PhD in computer science and master’s degrees in both computer science and business administration.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)

Contact us

For conference registration information and customer service

For more information on community discounts and trade opportunities with O’Reilly conferences

Become a sponsor

For information on exhibiting or sponsoring a conference

For media/analyst press inquires