In many software companies, A/B testing is a central technique for understanding how a product change will affect users. However, experiments are often run for a short time, with the assumption that the effects (or lack of measured effects) will persist over time. The users who are most likely to be in an experiment are those most likely to use the service, so the samples for many A/B tests are biased toward very engaged users.
For experiments specifically targeting new or less-engaged users, however, it is both harder to get a sample sufficient to measure effects and to understand how those effects will accumulate over time. For example, when testing out new copy for an email campaign, all users eligible to receive emails would be triggered in immediately; the result of the experiment should give an estimate of the impact for the whole population.
On the other hand, testing a new search experience for users who have been dormant for several months can only include users who have been dormant, visited the site, and then performed a search. This set of users may be quite small each week, and it may take many weeks to reach the entire population. Running such an experiment for only a few weeks is unlikely to have the power to detect a reasonably sized effect. Even if an effect is detected, the number of users in the experiment will seem low because it is difficult to estimate the number of users who are not yet in the experiment.
Brian Karfunkel details a method for estimating the total population that would be affected by a product change over the course of a quarter, developed at Pinterest. Using data on how many users enter the experiment each day, Pinterest fits models to predict the long-term population that would be affected if it shipped the change and uses that to extrapolate the long-term absolute impact. The tool Pinterest developed to implement this method allows the company to consider many types of experiments on an equal footing, ensuring that it does not just value the many users who come every week but also those who come less regularly.
Brian Karfunkel is a data scientist at Pinterest. Previously, he was senior data analyst at the NYU Furman Center, where he worked on housing and urban policy issues, and a research fellow at Stanford Law School, where he helped research the effects of workplace safety and health policy.
Comments on this page are now closed.
©2018, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org