A/B testing at scale: Accelerating software innovation

Ronny Kohavi is a Microsoft distinguished engineer and the general manager for the analysis and experimentation team within Microsoft’s Artificial Intelligence and Research Group. Previously, he was partner architect at Bing and founder of the experimentation platform team. Prior to Microsoft, he was the director of data mining and personalization at Amazon; the vice president of business intelligence at Blue Martini Software (acquired by Red Prairie); and manager of the MineSet project, Silicon Graphics’ award-winning product for data mining and visualization. Ronny was the general chair for KDD 2004, cochair of KDD 99’s industrial track with Jim Gray, and cochair of the KDD Cup 2000 with Carla Brodley and has been an invited or keynote speaker at a number of conferences around the world. His papers have over 34,000 citations; three of them are in the top 1,000 most-cited papers in computer science. In 2016, he was named the fifth-most-influential scholar in AI and the twenty-sixth most influential scholar in machine learning. Ronny holds a PhD in machine learning from Stanford University, where he led the MLC++ project (the machine learning library in C++ used in MineSet and at Blue Martini Software), and a BA from the Technion, Israel.

• Website

Alex Deng is a principal data scientist manager on Microsoft’s analysis and experimentation team, where he and his team work on methodological improvements of the experimentation platform as well as related engineering challenges. Alex has published his work in conference proceedings like KDD, WWW, WSDM, and other statistical journals. He colectured a tutorial on A/B testing at JSM 2015. Alex holds a PhD in statistics from Stanford University and a BS in mathematics from Zhejiang university.

Somit Gupta is a senior data scientist with Microsoft’s analysis and experimentation team. Recently, he helped the MSN and Edge browser content teams scale their experimentation and analyze those experiments with better OEC and diagnostic metrics. Somit holds a master’s degree in computer science from the University of Waterloo, Canada.

Paul Raff is a principal data scientist manager on Microsoft’s analysis and experimentation team, where he and his team work to enable scalable experimentation for teams around Microsoft, including Windows 10, Office Online, Exchange Online, and Cortana, focusing on experiment quality and ensuring that all experiments are operating as intended and in a way that allows for the appropriate conclusions to be made. Previously, he was a supply chain researcher at Amazon. Paul holds a PhD in mathematics from Rutgers University as well as degrees in mathematics and computer science from Carnegie Mellon University.

• Website