• Microsoft
  • Nebula
  • Google
  • SugarCRM
  • Facebook
  • HP
  • Intel
  • Rackspace Hosting
  • WSO2
  • Alfresco
  • BlackBerry
  • Dell
  • eBay
  • Heroku
  • InfiniteGraph
  • JBoss
  • LeaseWeb
  • Liferay
  • Media Temple, Inc.
  • OpenShift
  • Oracle
  • Percona
  • Puppet Labs
  • Qualcomm Innovation Center, Inc.
  • Rentrak
  • Silicon Mechanics
  • SoftLayer Technologies, Inc.
  • SourceGear
  • Urban Airship
  • Vertica
  • VMware
  • (mt) Media Temple, Inc.

Sponsorship Opportunities

For information on exhibition and sponsorship opportunities at the convention, contact Sharon Cordesse at scordesse@oreilly.com

Download the OSCON Sponsor/Exhibitor Prospectus

Contact Us

View a complete list of OSCON contacts

Building an A/B Testing Framework for Web Applications

Tools and Techniques
Location: Portland 255
Average rating: **...
(2.50, 2 ratings)

We’ll start with an overview of the roles that A/B testing can play in a web application (design optimization, feature ramp-up, backend migrations and others) and how it relates to other techniques (e.g., side-by-side testing). We’ll then outline the various approaches for implementing A/B testing and discuss their pros & cons.

In the second part of the talk, we’ll present one particular implementation, Etsy’s A/B framework (it’ll be open sourced by OSCON time) in detail. We’ll look at our PHP API and how we integrate it with a variety of other tools (Apache, Hadoop, Splunk…). We’ll include a few notes on experiment design along the way.

The last part of the tutorial will be hands-on: PHP developers can try out the Etsy framework; developers who are using or interested in other stacks can take the ideas from the talk and code up the rudiments of an A/B framework in their favorite language. We hope to take a look at some of these efforts and discuss how one might be able to build on them.

Zhi-Da Zhong


Zhi-Da is a software engineer who has worked in a variety of areas including parallel computing, security, and web applications. He’s built and used several A/B testing systems in recent years. He currently works on A/B testing, search, and other neat stuff at Etsy.

Comments on this page are now closed.


Raymond Wong
10/20/2011 5:49am PDT

Is the session recorded? I try to look for a youTube video but could not find it.

Zhi-Da Zhong
07/31/2011 1:53pm PDT

The presentation notes have been uploaded.

Zhi-Da Zhong
07/26/2011 9:25am PDT

Updated again with minor corrections and a few additional bullet points that were said but mistakenly left out of the original slides.

Zhi-Da Zhong
07/26/2011 6:38am PDT

The PDF has been updated.

Craig Gerhard
07/26/2011 4:20am PDT

The PDF listed above is not a true representation of the slides/examples used in the presentation. Many slides were missing/different and my assumption is that it was due to last minute changes. Can you update the PDF above when you get a chance? Thanks.