Building an A/B Testing Framework for Web Applications

Tools and Techniques
Location: Portland 255
Average rating: **...
(2.50, 2 ratings)

We’ll start with an overview of the roles that A/B testing can play in a web application (design optimization, feature ramp-up, backend migrations and others) and how it relates to other techniques (e.g., side-by-side testing). We’ll then outline the various approaches for implementing A/B testing and discuss their pros & cons.

In the second part of the talk, we’ll present one particular implementation, Etsy’s A/B framework (it’ll be open sourced by OSCON time) in detail. We’ll look at our PHP API and how we integrate it with a variety of other tools (Apache, Hadoop, Splunk…). We’ll include a few notes on experiment design along the way.

The last part of the tutorial will be hands-on: PHP developers can try out the Etsy framework; developers who are using or interested in other stacks can take the ideas from the talk and code up the rudiments of an A/B framework in their favorite language. We hope to take a look at some of these efforts and discuss how one might be able to build on them.

Zhi-Da Zhong


Zhi-Da is a software engineer who has worked in a variety of areas including parallel computing, security, and web applications. He’s built and used several A/B testing systems in recent years. He currently works on A/B testing, search, and other neat stuff at Etsy.

Comments on this page are now closed.


Raymond Wong
10/20/2011 5:49am PDT

Is the session recorded? I try to look for a youTube video but could not find it.

Zhi-Da Zhong
07/31/2011 1:53pm PDT

The presentation notes have been uploaded.

Zhi-Da Zhong
07/26/2011 9:25am PDT

Updated again with minor corrections and a few additional bullet points that were said but mistakenly left out of the original slides.

Zhi-Da Zhong
07/26/2011 6:38am PDT

The PDF has been updated.

Craig Gerhard
07/26/2011 4:20am PDT

The PDF listed above is not a true representation of the slides/examples used in the presentation. Many slides were missing/different and my assumption is that it was due to last minute changes. Can you update the PDF above when you get a chance? Thanks.