Presented By O'Reilly and Cloudera
Make Data Work
5–7 May, 2015 • London, UK

What's there to know about A/B testing?

Noel Welsh (Underscore Consulting)
14:35–14:55 Wednesday, 6/05/2015
Data Science
Location: King's Suite - Balmoral
Average rating: ***..
(3.78, 9 ratings)

Prerequisite Knowledge

A basic understanding of significance testing will be helpful.


The hypothesis test is the standard tool for A/B testing. Taught in every introductory statistics class, its familiarity if often mistaken for simplicity. There are hosts of issues with running with a meaningful test, ranging from methodological — the nuts and bolts of collecting and interpreting data — to epistemological — questioning the very foundations of significance testing. In this talk I will discuss some of these issues. My aim is to inspire debate within the community, leading to a more nuanced view of A/B testing and better practices.

The first part of my talk will discuss the meaning and methodology of the classic hypothesis test. We’ll cover fundamental assumptions behind A/B testing, type I and type II errors, and common mistakes like early stopping and misinterpreting results.

The second part will consider alternatives to the classic approach, confidence intervals and Bayesian methods, that still fit within the significance testing paradigm. I will describe how these alternatives give us additional information that allows more nuanced decision-making than the binary significant/not-significant approach often adopted with significance tests.

Finally, I’ll ask you to consider the basic underpinnings of significance tests. The goals of science (discovering truth) are not the goals of business (making money), and there are good reasons to look beyond the framework of significance testing in the business world. I’ll describe scenarios where the fundamental assumptions of significance testing break down, and briefly discuss some of the alternatives approaches we can use in its place.

Photo of Noel Welsh

Noel Welsh

Underscore Consulting

Noel has over 15 years experience in software architecture and development, and over a decade in machine learning and data mining. His current project is Myna, which makes bandit algorithms accessible to all. Previous projects he’s been involved with include one of the first commercial products to apply machine learning to the internet (eventually acquired by Omniture), a BAFTA-award winning website, and a custom CMS used daily by thousands of students.

Noel is an active writer, presenter, and open source contributor. Noel has a PhD in machine learning from the University of Birmingham.