Data driving decisioning is not an invention of the recent past. We have a waste collection of metrics, KPIs, and other statistics that help decision makers to be more data driven. While the recent technological advances have increased our ability to see the world through the objective lenses of data, one unintended side effect of exactly this technology is the diminishing value of many metrics we cherish.
Consider the clickthrough rate (CTR) in advertising. With the rise of Doubleclick, CTR has been a core metric of an advertiser’s ability to identify a relevant audience: more clicks equals more interest equals more qualified potential consumers. While this might have been true in the past, the sad reality is that measures like CTR have lost their meaning.
The issue is not as much adversarial attempts of gaming, but the fact that even highly correlated proxies like CTR lose their power once optimization can draw from more granular information. Given much more detailed information, modern optimization approaches can find signals in the noise, and identify users with poor vision or bad fine motor skills who accidentally (but not randomly) click on ads, rather than interested potential consumers. This fundamental issue extends beyond clicks and calls for a close examination of many of our favorite KPIs.
Prior to joining Dstillery (former Media6Degrees), Claudia Perlich spent five years working at the Data Analytics Research group at the IBM T.J. Watson Research Center, concentrating on research in data analytics and machine learning for complex real-world domains and applications. She has been published in over 30 scientific publications and holds multiple patents in the area of machine learning. Claudia has won many data mining competitions, including the prestigious 2007 KDD CUP on movie ratings, the 2008 KDD CUP on breast-cancer detection, and the 2009 KDD CUP on churn and propensity predictions for telecommunication customers. Claudia received her Ph.D. in Information Systems from Stern School of Business, New York University in 2005, and holds a Master of Computer Science from Colorado University.
©2015, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com
Apache Hadoop, Hadoop, Apache Spark, Spark, and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries, and are used with permission. The Apache Software Foundation has no affiliation with and does not endorse, or review the materials provided at this event, which is managed by O'Reilly Media and/or Cloudera.