Making Open Work
May 8–9, 2017: Training & Tutorials
May 10–11, 2017: Conference
Austin, TX

A/B testing at scale: Developing an in-house A/B testing framework for big testing and big data

Mita Mahadevan (Intuit)
1:45pm2:25pm Wednesday, May 10, 2017
Data, Big and Small
Location: Meeting Room 18 C/D
Level: Intermediate
Average rating: ***..
(3.64, 11 ratings)

Who is this presentation for?

  • Data scientists, data engineers, and executives who are seeking to engender a data-driven mentality companywide

Prerequisite knowledge

  • A working knowledge of A/B testing and open source concepts

What you'll learn

  • Learn about the top challenges of A/B testing and the key attributes that solutions need for teams to be successful
  • Understand how to move from using data primarily as a measurement tool to using data to better understand customers’ behaviors through fast data-driven experimentation
  • Explore the A/B testing tool landscape (commercial tools versus open source tools versus in-house tools)
  • Discover an open source framework available for diverse product testing use cases
  • Learn the platform architecture and workflows to go from data collection to analysis using Hadoop and other big data technologies


A/B testing is the norm when it comes to rapid product iteration, yet many companies still struggle to realize its full benefits. How do you scale A/B testing companywide? Beyond marketing and design, how do you use this powerful technique to determine in-product features, assess algorithms, and ultimately develop deep personalized experiences and artificial intelligence?

To address these challenges, many leading tech companies (Uber, Netflix, LinkedIn, and others) have recently decided to build their own scalable, in-house product-testing data platforms from the ground up to enable experimentation and engender a data-driven mentality across different products.

Mita Mahadevan explores why and how companies are developing in-house A/B testing frameworks and offers guidance from the front lines on dos and don’ts for those in the midst of their journey to become data-driven.

Mita begins by sharing a methodology to design experiments that has been proven to be effective across industries, highlighting use cases across social networking, music streaming, financial services, and gaming with varying metrics and optimization targets using insights derived from a variety of verticals. Mita then offers a deep dive into Intuit’s implementation of an in-house A/B testing framework, which is available as open source software. Intuit faced many obstacles in scaling its framework, including integrating across 64+ products with fragmented data in silos and unifying testing across mobile, web, desktop, frontend, and backend. Mita explains how the team solved for different applications, as well as how other companies like LinkedIn, Netflix, and Uber are tackling similar challenges using various open source technologies.

Photo of Mita Mahadevan

Mita Mahadevan


Mita Mahadevan leads the development of data products at Intuit’s Data Engineering and Analytics (IDEA) group. Mita started her career building distributed analytic systems to analyze billions of retail transactions. Her experience spans several domains, from retail analytics at Demandtec (IBM) to social network analysis at Ning. Some notable data products she has helped build include automated attribution for retail pricing, detecting growth, and diffusion patterns in online communities. Mita mentors and advises students at Hackbright and a few of the big data fellowship programs and has presented at the GHC and other industry meetups and conferences. Her hobbies include applying management principles to parenting her twin boys.