Machine learning models are often complex, with massive abstract descriptions that make the relationship between their inputs and outputs seem like a black box. A modern neural network, for example, might look at thousands of features and perform millions of additions and multiplications to produce a prediction. But how do we explain that prediction to someone else? How do we tell which features are important and why? And if we can’t understand how a model makes a prediction, do we really trust it to run our business, make medical conclusions, or make an unbiased decision about an applicant’s eligibility for a loan?
Explainability techniques clarify how models make decisions, offering answers to these questions and giving us confidence that our models are functioning properly (or not). Each of these techniques is applicable to a different set of models, makes different assumptions, and answers a slightly different question, but when used properly, they can meet business requirements and improve model performance.
Mike Ruberry offers an overview of the two main types of explainability. The first directly relates inputs to outputs, a naturally intuitive approach that includes local interpretable model-agnostic explanations (LIME), axiomatic attributions, VisualBackProp, and traditional feature contributions. The second makes use of the data the model was trained on. DeepLift, for example, can show which training examples were most relevant to a model’s decision, while scrambling and prototype methods detail the decision making process. Along the way, Mike covers how ZestFinance approaches explainability, offering a practical guide for your own work. While there is no perfect silver-bullet explainability technique, understanding when and how to use these approaches will let you explain many useful models and give you a broad view of current explainability best practices and research.
Evan Kriminger is a Senior Associate of Data Science at ZestFinance, where his research interests include explainability and building efficient tools for training deep neural networks. He holds a PhD from the Computational NeuroEngineering Laboratory at the University of Florida, completing a dissertation on active learning and constrained clustering. Prior to ZestFinance, he worked at Leap Motion, conducting machine learning research for hand tracking.
©2018, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com