Skip to main content

How to Get Statistics Right in AB Testing: The Short Answer (With Proof from Four Years of Fundraising Data from Wikipedia)

Zack Exley (Brand New Congress), Sahar Massachi (Independent)
Data Science Beekman Parlor - Sutton North
Average rating: ****.
(4.00, 13 ratings)
Slides:   1-PPTX    2-PDF 

The Wikipedia fundraising team was performing up to 100 AB tests per week. It wasn’t enough to find the gains we needed. We needed to use statistics to interpret our AB tests accurately, but also to estimate smallest acceptable sample sizes to increase testing frequency. We were not comfortable trusting methods proposed by other practitioners or academics who could not prove that their methods would work accurately for our data.

We’ll present simple methods that we believe accurately predict future performance from AB test results, and that allow us to determine the smallest acceptable sample size. Using four years of AB testing data, we’ll show that these methods really work. By taking a walk through four years of data, we’ll show how our current methods would have predicted the actual outcomes of long tests with much smaller samples, and the implications of various decisions about statistical confidence, power, sample size and choice of statistical method.

This session will be useful to anyone who performs AB tests, manages teams that rely on AB testing, or holds responsibility for implementing or maintaining AB testing platforms.

Photo of Zack Exley

Zack Exley

Brand New Congress

Zack Exley is the Chief Revenue Officer at the Wikimedia Foundation. He has recently worked at ThoughtWorks, a global IT consultancy and OMP, a U.S. fundraising and communications firm. He is the Co-Founder and President of the New Organizing Institute, a technology and strategy training center for social justice organizations.

Zack helped bring U.S. politics online as’s first Organizing Director, advisor to the Howard Dean campaign, and Director of Online Communications and Organizing at John Kerry’s presidential campaign. He often appeared as a commentator on national and international media and his work was frequently profiled.

He began developing tools for online political organizing in the 2000 U.S. election controversy, when he used a website to allow citizens to self-organize more than 100 rallies around the United States. He began his political career working as a union organizer, and has also worked as a computer programmer, factory worker, short order cook, book binder and janitor. He lives in rural Missouri with his wife and daughter.

Sahar Massachi


Sahar is a startup founder, online organizer, and software developer recently out of Brandeis with a BS and MA in Computer Science. After founding a start up to provide community organizations with better mobile tools, he joined up with the Wikipedia fundraising team to help complete a review of their AB testing methods, analyzing and preparing for publication hundreds of past AB tests.

Comments on this page are now closed.


Marek K Kolodziej
10/30/2013 4:30pm EDT

Would it be possible to post the slides here, like the other speakers have?


Sponsorship Opportunities

For exhibition and sponsorship opportunities, contact Susan Stewart at

Media Partner Opportunities

For information on trade opportunities with O'Reilly conferences email mediapartners

Press & Media

For media-related inquiries, contact Maureen Jennings at

Contact Us

View a complete list of Strata + Hadoop World 2013 contacts