Mar 15–18, 2020

Getting the most out of your AI projects with model feedback loops

Hannes Hapke (SAP ConcurLabs), Catherine Nelson (Concur Labs, SAP Concur)
11:00am11:40am Tuesday, March 17, 2020
Location: 210 C/G

Who is this presentation for?

  • Data scientists, machine learning engineers, and data science team managers




Building machine learning models consumes a considerable amount of time, money and energy. But much of the investment is lost if your model performance isn’t measured and monitored. While many people talk about how to deploy a model to production, very few discussions cover how to measure the model’s performance. Hannes Hapke and Catherine Nelson want to change that by introducing model feedback loops.

Model feedback loops have two important purposes: measure the model’s performance and increase the model’s underlying data set for future model updates.

You’ll explore how you can set up feedback loops, measure your model’s performance, and how to use feedback loops to retrain you models. They introduce metrics to measure your model’s performance, highlight user experience considerations when designing model feedback loops, and outline a framework for building reproducible feedback loops.

Prerequisite knowledge

  • Familiarity with basic machine learning concepts and at least one machine learning framework such as PyTorch, TensorFlow, or Keras (The examples are based on TensorFlow and Keras, but core concepts can be applied to any framework.)

What you'll learn

  • Discover what feedback loops are in the context of machine learning, why they're important, and what UX considerations should be applied when designing feedback loops
  • Learn how to measure a model’s performance, the ROI of the data science project, and which KPIs to choose; how explicit and implicit feedback differ; and how to efficiently track your model’s performance and how to turn it into new datasets to generate new model versions
Photo of Hannes Hapke

Hannes Hapke

SAP ConcurLabs

Hannes Hapke is a senior data scientist at SAP ConcurLabs. He’s been a machine learning enthusiast for many years and is a Google Developer Expert for machine learning. Hannes has applied deep learning to a variety of computer vision and natural language problems, but his main interest is in machine learning infrastructure and automating model workflows. Hannes is a coauthor of the deep learning publication Natural Language Processing in Action and he’s currently working on a book about TensorFlow Extended Building Machine Learning Pipelines (O’Reilly). When he isn’t working on a deep learning project, you’ll find him outdoors running, hiking, or enjoying a good cup of coffee with a great book.

Photo of Catherine Nelson

Catherine Nelson

Concur Labs, SAP Concur

Catherine Nelson is a senior data scientist for Concur Labs at SAP Concur, where she explores innovative ways to use machine learning to improve the experience of a business traveller. She’s particularly interested in privacy-preserving ML and applying deep learning to enterprise data. Previously, she was a geophysicist and studied ancient volcanoes and explored for oil in Greenland. Catherine has a PhD in geophysics from Durham University and a master’s of earth sciences from Oxford University.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)

Contact us

For conference registration information and customer service

For more information on community discounts and trade opportunities with O’Reilly conferences

Become a sponsor

For information on exhibiting or sponsoring a conference

For media/analyst press inquires