A/B testing is the norm when it comes to rapid product iteration, yet many companies still struggle to realize its full benefits. How do you scale A/B testing companywide? Beyond marketing and design, how do you use this powerful technique to determine in-product features, assess algorithms, and ultimately develop deep personalized experiences and artificial intelligence?
To address these challenges, many leading tech companies (Uber, Netflix, LinkedIn, and others) have recently decided to build their own scalable, in-house product-testing data platforms from the ground up to enable experimentation and engender a data-driven mentality across different products.
Mita Mahadevan explores why and how companies are developing in-house A/B testing frameworks and offers guidance from the front lines on dos and don’ts for those in the midst of their journey to become data-driven.
Mita begins by sharing a methodology to design experiments that has been proven to be effective across industries, highlighting use cases across social networking, music streaming, financial services, and gaming with varying metrics and optimization targets using insights derived from a variety of verticals. Mita then offers a deep dive into Intuit’s implementation of an in-house A/B testing framework, which is available as open source software. Intuit faced many obstacles in scaling its framework, including integrating across 64+ products with fragmented data in silos and unifying testing across mobile, web, desktop, frontend, and backend. Mita explains how the team solved for different applications, as well as how other companies like LinkedIn, Netflix, and Uber are tackling similar challenges using various open source technologies.
Mita Mahadevan leads the development of data products at Intuit’s Data Engineering and Analytics (IDEA) group. Mita started her career building distributed analytic systems to analyze billions of retail transactions. Her experience spans several domains, from retail analytics at Demandtec (IBM) to social network analysis at Ning. Some notable data products she has helped build include automated attribution for retail pricing, detecting growth, and diffusion patterns in online communities. Mita mentors and advises students at Hackbright and a few of the big data fellowship programs and has presented at the GHC and other industry meetups and conferences. Her hobbies include applying management principles to parenting her twin boys.
©2017, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org