The Wikipedia fundraising team was performing up to 100 AB tests per week. It wasn’t enough to find the gains we needed. We needed to use statistics to interpret our AB tests accurately, but also to estimate smallest acceptable sample sizes to increase testing frequency. We were not comfortable trusting methods proposed by other practitioners or academics who could not prove that their methods would work accurately for our data.
We’ll present simple methods that we believe accurately predict future performance from AB test results, and that allow us to determine the smallest acceptable sample size. Using four years of AB testing data, we’ll show that these methods really work. By taking a walk through four years of data, we’ll show how our current methods would have predicted the actual outcomes of long tests with much smaller samples, and the implications of various decisions about statistical confidence, power, sample size and choice of statistical method.
This session will be useful to anyone who performs AB tests, manages teams that rely on AB testing, or holds responsibility for implementing or maintaining AB testing platforms.
Zack Exley is the Chief Revenue Officer at the Wikimedia Foundation. He has recently worked at ThoughtWorks, a global IT consultancy and OMP, a U.S. fundraising and communications firm. He is the Co-Founder and President of the New Organizing Institute, a technology and strategy training center for social justice organizations.
Zack helped bring U.S. politics online as MoveOn.org’s first Organizing Director, advisor to the Howard Dean campaign, and Director of Online Communications and Organizing at John Kerry’s presidential campaign. He often appeared as a commentator on national and international media and his work was frequently profiled.
He began developing tools for online political organizing in the 2000 U.S. election controversy, when he used a website to allow citizens to self-organize more than 100 rallies around the United States. He began his political career working as a union organizer, and has also worked as a computer programmer, factory worker, short order cook, book binder and janitor. He lives in rural Missouri with his wife and daughter.
Sahar is a startup founder, online organizer, and software developer recently out of Brandeis with a BS and MA in Computer Science. After founding a start up to provide community organizations with better mobile tools, he joined up with the Wikipedia fundraising team to help complete a review of their AB testing methods, analyzing and preparing for publication hundreds of past AB tests.
Comments on this page are now closed.
For exhibition and sponsorship opportunities, contact Susan Stewart at firstname.lastname@example.org
For information on trade opportunities with O'Reilly conferences email mediapartners
For media-related inquiries, contact Maureen Jennings at email@example.com
View a complete list of Strata + Hadoop World 2013 contacts