The Day Zach Galifianakis Saved Healthcare: Impact and Attribution in Data Science
Data and experiment driven cultures are steadily growing in the tech industry. While fostering such a culture reaps many benefits for a company it also brings an important mandate to properly instrument, measure, and attribute experiment impact. While the gold standard of A/B testing allows for straightforward experimental analysis, there are a number of scenarios that are not amenable to A/B testing due to various constraints (financial feasibility, technical capability, etc.).
Such “non-standard” quasi-experimental events are quite common but many companies, even with data driven cultures, ignore them since they fall outside the randomized control trial framework. In this talk we will explore a number of techniques that allow for improved impact measurement and attribution that enhance each other either in an iterative or modular way that allow data scientists to derive value from what might normally be thought of as “messy” or “unusable” data.
We will learn about these techniques with the aid of examples from the popular press (Zach Galifianakis and healthcare.gov), Microsoft advertising (television and print), and Bing experimentation (comparisons of A/B tests and techniques outlined in this talk). In each case we will compare analysis techniques, point out inconsistencies in naive analysis, and build methods to avoid such mistakes.
The goal of this talk is for the audience to not only gain an understanding of why impact and attribution are important, but also to understand the assumptions, pit falls, and strengths of various analytic approaches to dealing with impact and attribution. This talk is intended to bridge the gap from initial instrumentation, infrastructure, and dash boarding to designing experiments that move metrics in a positive way and understanding what caused them to move in the first place.
Chris Harland is a Data Scientist at Microsoft working on problems in Bing search, Windows, and MSN. He holds a PhD in Physics from the University of Oregon and has worked in a wide variety of fields spanning elementary science education, cutting edge biophysical research, and recommendation/personalization engines.
Ever since Chris started using Bayesian methods on a semi-regular basis the frequency with which he uses the phrase “well, maybe” in conversation with colleagues has increased ten fold. His colleagues have yet to forgive Bayes for this.