Sep 9–12, 2019

Talking to the Machines: Monitoring production machine learning systems

Ting-Fang Yen (DataVisor)
4:00pm4:40pm Thursday, September 12, 2019
Location: Expo Hall 3

Who is this presentation for?

Data scientists and engineers that build and/or maintain production machine learning models.




Production machine learning systems require constant monitoring, not just to keep the system online, but also to ensure the model inference results are correct. This is much more straightforward when user feedback or labels are available. In those cases, the model performance can be tracked and periodically re-evaluated using standard metrics such as precision, recall, or AUC. But what about when labeled data is lacking? In many applications, labels are expensive to obtain (requiring human analysts’ manual review) or cannot be obtained in a timely manner (e.g., not available until weeks or months later).

In this talk, we describe the design and implementation of a real-time system to monitor production machine learning systems. Our approach is designed to discover detection anomalies, such as volume spikes caused by spurious false positives, as well as gradual concept drifts when the model is no longer able to capture the target concept. In either case, we are able to automatically detect undesirable model behaviors early.

A part of our approach borrows from signal processing techniques for time series decomposition, where the time series can be used to represent a sequence of model decisions on different types of input data, or the amount of deviation between consecutive model runs. We calculate cross-correlation among the identified anomalies to facilitate root cause analysis of the model behavior.

This work is a step toward automated deployment of machine learning in production as well as new tools for interpreting model inference results.

Prerequisite knowledge

Audience members should have a general understanding of machine learning technologies, and some experience of working with large-scale datasets and systems.

What you'll learn

1. An understanding the practical challenges of deploying machine learning systems 2. Experiences from maintaining a production machine learning system that handles over a billion requests per day on average 3. Metrics for model quality when labeled data is not available
Photo of Ting-Fang Yen

Ting-Fang Yen


Ting-Fang Yen is director of research at DataVisor, the leading fraud detection platform powered by transformational AI technology. Her work applies big data analytics and machine learning to tackle problems in cybersecurity. Ting-Fang holds a PhD in electrical and computer engineering from Carnegie Mellon University.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)

Contact us

For conference registration information and customer service

For more information on community discounts and trade opportunities with O’Reilly conferences

Become a sponsor

For information on exhibiting or sponsoring a conference

Contact list

View a complete list of O'Reilly AI contacts