• Intel
  • Microsoft
  • Google
  • Sun Microsystems
  • BT
  • IBM
  • Yahoo! Inc.
  • Zimbra
  • Atlassian Software Systems
  • Disney
  • EnterpriseDB
  • Etelos
  • Ingres
  • JasperSoft
  • Kablink
  • Linagora
  • MindTouch
  • Mozilla Corporation
  • Novell, Inc.
  • Open Invention Network
  • OpSource
  • RightScale
  • Silicon Mechanics
  • Tenth Planet
  • Ticketmaster
  • Voiceroute
  • White Oak Technologies, Inc.
  • XAware
  • ZDNet

Sponsorship Opportunities

For information on exhibition and sponsorship opportunities at the conference, contact Sharon Cordesse at scordesse@oreilly.com.

Media Partner Opportunities

Download the Media & Promotional Partner Brochure (PDF) for more information on trade opportunities with O'Reilly conferences, or contact mediapartners@oreilly.com.

Press and Media

For media-related inquiries, contact Maureen Jennings at maureen@oreilly.com.

OSCON Newsletter

To stay abreast of conference news and to receive email notification when registration opens, please sign up for the OSCON newsletter (login required).

Contact Us

View a complete list of OSCON 2008 Contacts

Effective A/B Testing

Tutorial, Web Applications
Location: D136 Level: Novice
Presentation: external link
Average rating: ***..
(3.67, 12 ratings)

A/B testing is a simple idea. Create two versions of something. Put people through the different versions. Track what they do afterwards. Figure out which version causes more people to do what you want. Go with the better version. Iterate.

The web is a perfect fit for this methodology. It is easy to have multiple versions of a web page. We know how to track users. We often want to know what causes users to react, but don’t.

The potential wins are huge. The first successful A/B test I was involved with grew the company by 40%. I have seen many changes to emails or forms change response rates by 5-10% or more. In either direction.

But there are challenges when trying to A/B test. Suppose you run a test and one group does better. Are we confident that the improvement is real and not just chance? How do we tell? Can you test multiple things at once? What best practices and potential pitfalls do you need to be aware of?

This tutorial will answer those questions and more. Afterwards you should be able to set up, run, and evaluate A/B tests, including being able to perform the calculations that you need to evaluate the answers.

Ben Tilly


Ben Tilly has been active in the Perl community for many years, and is currently on The Perl Foundation grant committee. He has been running A/B tests since early 2004, and has helped several companies in the Los Angeles area set up A/B testing. He is currently the Reporting Architect at Pictage, Inc where his responsibilities include A/B testing.

OSCON 2008