Using ML in real-world applications and production systems is a very complex task involving issues rarely encountered in toy problems, R&D environments, or offline cases. Key considerations for accessing the decay, current status, and production readiness of ML systems include testing, monitoring, and logging, but how much is enough? It’s difficult to know where to get started or even to know who should be responsible for the testing and monitoring. If you’ve heard the phrase “test in production” too often when it comes to ML, perhaps you need to change your strategy.
Tania Allard dives deep into some of the most frequent issues encountered in real-life ML applications and how you can make your systems more robust, and she explores a number of indicators pointing to decay of models or algorithms in production systems. Some of the topics covered include problems and pitfalls of ML in production; introducing a rubric to test and monitor your ML applications; and testing data and features, testing your model development, monitoring your ML applications, and model decay.
You’ll leave with a clear rubric with actionable tests and examples to ensure the quality or model in production is adequate. Engineers, DevOps, and data scientists will gain valuable guidelines to evaluate and improve the quality of their ML models before anything reaches production stage.
Tania Allard (she/her) is a cloud developer advocate at Microsoft and a research engineer with vast experience in academic research and industrial environments. Her main areas of expertise are within data-intensive applications, scientific computing, and machine learning; one of her main areas is the improvement of processes, reproducibility, and transparency in research, data science, and artificial intelligence. Over the last few years, she’s trained hundreds of people on scientific computing, reproducible workflows, and ML models testing, monitoring, and scaling and delivered talks on the topic worldwide. She’s passionate about mentoring, open source and its community, and she’s involved in a number of initiatives aimed to build more diverse and inclusive communities. She’s also a contributor, maintainer, and developer of a number of open source projects and the Founder of Pyladies NorthWest UK.
©2019, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org