Presented By O’Reilly and Cloudera
Make Data Work
March 5–6, 2018: Training
March 6–8, 2018: Tutorials & Conference
San Jose, CA

A/B testing at scale: Accelerating software innovation

Ronny Kohavi (Microsoft), Alex Deng (Microsoft), Somit Gupta (Microsoft), Paul Raff (Microsoft)
1:30pm5:00pm Tuesday, March 6, 2018
Average rating: ****.
(4.00, 3 ratings)

Who is this presentation for?

  • Data scientists, software developers, and program managers

What you'll learn

  • Understand why running controlled experiments is critical to establish causality and why you should test everything with A/B tests


The internet provides developers of connected software, such as websites, applications, and devices, an unprecedented opportunity to accelerate innovation by evaluating ideas quickly and accurately using controlled experiments, also known as A/B tests. Search engines, retailers, social networking services, travel services, and startups of all stripes now use online controlled experiments to make data-driven decisions at a wide range of companies in everything from frontend user-interface changes to backend algorithms.

The theory of a controlled experiment is simple and dates back to Ronald A. Fisher’s experiments at the Rothamsted Agricultural Experimental Station in England in the 1920s. However, the deployment and evaluation of online controlled experiments at scale (think hundreds of concurrently running experiments) across variety of websites, mobile apps, and desktop applications presents many pitfalls and new research challenges.

Ronny Kohavi, Alex Deng, Somit Gupta, and Paul Raff lead an introduction to A/B testing and share lessons learned from one of the largest A/B testing platforms on the planet, running at Microsoft, which executes over 10K experiments a year. You’ll discover practical and research challenges in scaling experimentation and promising directions for future work.

Photo of Ronny Kohavi

Ronny Kohavi


Ronny Kohavi is a Microsoft distinguished engineer and the general manager for the analysis and experimentation team within Microsoft’s Artificial Intelligence and Research Group. Previously, he was partner architect at Bing and founder of the experimentation platform team. Prior to Microsoft, he was the director of data mining and personalization at Amazon; the vice president of business intelligence at Blue Martini Software (acquired by Red Prairie); and manager of the MineSet project, Silicon Graphics’ award-winning product for data mining and visualization. Ronny was the general chair for KDD 2004, cochair of KDD 99’s industrial track with Jim Gray, and cochair of the KDD Cup 2000 with Carla Brodley and has been an invited or keynote speaker at a number of conferences around the world. His papers have over 34,000 citations; three of them are in the top 1,000 most-cited papers in computer science. In 2016, he was named the fifth-most-influential scholar in AI and the twenty-sixth most influential scholar in machine learning. Ronny holds a PhD in machine learning from Stanford University, where he led the MLC++ project (the machine learning library in C++ used in MineSet and at Blue Martini Software), and a BA from the Technion, Israel.

Photo of Alex Deng

Alex Deng


Alex Deng is a principal data scientist manager on Microsoft’s analysis and experimentation team, where he and his team work on methodological improvements of the experimentation platform as well as related engineering challenges. Alex has published his work in conference proceedings like KDD, WWW, WSDM, and other statistical journals. He colectured a tutorial on A/B testing at JSM 2015. Alex holds a PhD in statistics from Stanford University and a BS in mathematics from Zhejiang university.

Photo of Somit Gupta

Somit Gupta


Somit Gupta is a senior data scientist with Microsoft’s analysis and experimentation team. Recently, he helped the MSN and Edge browser content teams scale their experimentation and analyze those experiments with better OEC and diagnostic metrics. Somit holds a master’s degree in computer science from the University of Waterloo, Canada.

Photo of Paul Raff

Paul Raff


Paul Raff is a principal data scientist manager on Microsoft’s analysis and experimentation team, where he and his team work to enable scalable experimentation for teams around Microsoft, including Windows 10, Office Online, Exchange Online, and Cortana, focusing on experiment quality and ensuring that all experiments are operating as intended and in a way that allows for the appropriate conclusions to be made. Previously, he was a supply chain researcher at Amazon. Paul holds a PhD in mathematics from Rutgers University as well as degrees in mathematics and computer science from Carnegie Mellon University.

Comments on this page are now closed.


03/09/2018 11:33pm PST

>Works for me.

Thanks Ronny, this URL works for me.

Clicking on the “View Slides” link above launches a blocking Safari warning stating: “This connection is not private” "This website may be impersonating “”…"

Picture of Ronny Kohavi
03/09/2018 4:16pm PST

> Link to the slides are not working. can you pl. check?
Works for me.

03/09/2018 10:07am PST

enjoyed very much. Link to the slides are not working. can you pl. check?

Picture of Paul Raff
03/08/2018 7:30am PST

Thanks, Kevin! We’re glad you liked it. We have the slides posted on our website,

03/08/2018 7:13am PST

Gents, really enjoyed this one. Can you post the slides? Thanks!