Are we doing this wrong? Advertisement features A/B testing

Description

A/B testing is the foundation of data-driven decision making. In today’s world, advertising is crucial to a website’s revenue, so it is even more important to measure the effects of changes correctly. However, A/B testing frameworks and systems are a double-edged sword. On one hand, you can leverage a scale-ready product, but on the other, testing could hide some statistical magic that, along with implementation flaws, might threaten the validity of your test results.

Chen Salomon demonstrates how to correctly design and implement an advertisement A/B testing and shares pitfalls, potential biases related to advertisement metrics, and possible mitigations. Along the way, you’ll explore more advanced methods such as multi-armed bandits, which can reduce the negative revenue loss due to “bad” options (condition groups), and take a closer look at advertising-specific metrics such as fill rates and viewability. Chen also discusses some real-world use cases of analysts and test designers ignoring the potential bias those parameters introduce to the results. Chen ends by offering practical tools, such as two-step allocation, to mitigate potential bias cases both in the test design phase and at the test runtime.