A/B testing is the foundation of data-driven decision making. In today’s world, advertising is crucial to a website’s revenue, so it is even more important to measure the effects of changes correctly. However, A/B testing frameworks and systems are a double-edged sword. On one hand, you can leverage a scale-ready product, but on the other, testing could hide some statistical magic that, along with implementation flaws, might threaten the validity of your test results.
Chen Salomon demonstrates how to correctly design and implement an advertisement A/B testing and shares pitfalls, potential biases related to advertisement metrics, and possible mitigations. Along the way, you’ll explore more advanced methods such as multi-armed bandits, which can reduce the negative revenue loss due to “bad” options (condition groups), and take a closer look at advertising-specific metrics such as fill rates and viewability. Chen also discusses some real-world use cases of analysts and test designers ignoring the potential bias those parameters introduce to the results. Chen ends by offering practical tools, such as two-step allocation, to mitigate potential bias cases both in the test design phase and at the test runtime.
Chen Salomon is the architect at high-scale storytelling platform Playbuzz, where, as the first employee, he has been responsible for the design and implementation of a scale-ready system since day one and implemented Playbuzz’s data pipeline, which collects, enriches, and stores thousands of events per second. An experienced developer, Chen specializes in high-scale web environments, specifically caching, CDN, cloud architectures, and microservices architectures.
Chen’s academic background includes research in the fields of social networks and content distribution with a focus on online experiments.
Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?
Join the conversation here (requires login)
©2018, O’Reilly UK Ltd • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org