Presented By O’Reilly and Cloudera
Make Data Work
21–22 May 2018: Training
22–24 May 2018: Tutorials & Conference
London, UK

Are we doing this wrong? Advertisement features A/B testing

Chen Salomon (Playbuzz)
14:5515:35 Thursday, 24 May 2018
Data science and machine learning, Data-driven business management
Location: Capital Suite 14 Level: Intermediate
Average rating: ****.
(4.00, 1 rating)

Who is this presentation for?

  • Those responsible for design A/B tests

Prerequisite knowledge

  • A basic understanding of statistics, A/B testing measurement, and advertising measurement concepts

What you'll learn

  • Understand possible bias factors in A/B test on ads
  • Learn how to correctly design and measure an A/B test when having outer control (fill rates for example) and use multi-armed bandits to reduce test effects on users


A/B testing is the foundation of data-driven decision making. In today’s world, advertising is crucial to a website’s revenue, so it is even more important to measure the effects of changes correctly. However, A/B testing frameworks and systems are a double-edged sword. On one hand, you can leverage a scale-ready product, but on the other, testing could hide some statistical magic that, along with implementation flaws, might threaten the validity of your test results.

Chen Salomon demonstrates how to correctly design and implement an advertisement A/B testing and shares pitfalls, potential biases related to advertisement metrics, and possible mitigations. Along the way, you’ll explore more advanced methods such as multi-armed bandits, which can reduce the negative revenue loss due to “bad” options (condition groups), and take a closer look at advertising-specific metrics such as fill rates and viewability. Chen also discusses some real-world use cases of analysts and test designers ignoring the potential bias those parameters introduce to the results. Chen ends by offering practical tools, such as two-step allocation, to mitigate potential bias cases both in the test design phase and at the test runtime.

Photo of Chen Salomon

Chen Salomon


Chen Salomon is the architect at high-scale storytelling platform Playbuzz, where, as the first employee, he has been responsible for the design and implementation of a scale-ready system since day one and implemented Playbuzz’s data pipeline, which collects, enriches, and stores thousands of events per second. An experienced developer, Chen specializes in high-scale web environments, specifically caching, CDN, cloud architectures, and microservices architectures.
Chen’s academic background includes research in the fields of social networks and content distribution with a focus on online experiments.

Comments on this page are now closed.


Picture of Chen Salomon
Chen Salomon | ARCHITECT
1/06/2018 7:19 BST

Hi, I’ll check with the Strata team when they are going to post the slides.

31/05/2018 22:24 BST

Thanks again for the talk Chen, I’m looking forward to implementing the bandit method in our upcoming tests. Will you be posting your slides?