The hypothesis test is the standard tool for A/B testing. Taught in every introductory statistics class, its familiarity if often mistaken for simplicity. There are hosts of issues with running with a meaningful test, ranging from methodological — the nuts and bolts of collecting and interpreting data — to epistemological — questioning the very foundations of significance testing. In this talk I will discuss some of these issues. My aim is to inspire debate within the community, leading to a more nuanced view of A/B testing and better practices.
The first part of my talk will discuss the meaning and methodology of the classic hypothesis test. We’ll cover fundamental assumptions behind A/B testing, type I and type II errors, and common mistakes like early stopping and misinterpreting results.
The second part will consider alternatives to the classic approach, confidence intervals and Bayesian methods, that still fit within the significance testing paradigm. I will describe how these alternatives give us additional information that allows more nuanced decision-making than the binary significant/not-significant approach often adopted with significance tests.
Finally, I’ll ask you to consider the basic underpinnings of significance tests. The goals of science (discovering truth) are not the goals of business (making money), and there are good reasons to look beyond the framework of significance testing in the business world. I’ll describe scenarios where the fundamental assumptions of significance testing break down, and briefly discuss some of the alternatives approaches we can use in its place.
Noel has over 15 years experience in software architecture and development, and over a decade in machine learning and data mining. His current project is Myna, which makes bandit algorithms accessible to all. Previous projects he’s been involved with include one of the first commercial products to apply machine learning to the internet (eventually acquired by Omniture), a BAFTA-award winning website, and a custom CMS used daily by thousands of students.
Noel is an active writer, presenter, and open source contributor. Noel has a PhD in machine learning from the University of Birmingham.
©2015, O’Reilly UK Ltd • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org
Apache Hadoop, Hadoop, Apache Spark, Spark, and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries, and are used with permission. The Apache Software Foundation has no affiliation with and does not endorse, or review the materials provided at this event, which is managed by O'Reilly Media and/or Cloudera.