Presented By
O’Reilly + Cloudera
Make Data Work
March 25-28, 2019
San Francisco, CA
Please log in

Testing ad content with survey experiments

Patrick Miller (Civis Analytics)
3:50pm4:30pm Thursday, March 28, 2019
Average rating: ***..
(3.40, 5 ratings)

Who is this presentation for?

  • Data scientists and data scientist managers



What you'll learn

  • Understand how testing ad content before it is shown to an audience can prevent backlash
  • Explore methods for survey data collection, analysis, weighting, and presenting results to decision makers


Brands spend lots of money on ads but rely on intuition to choose the ad to show to an audience and avoid ads that cause backlash. Data scientists can help marketing departments optimize their decision making by testing the content of ads before they are shown to audiences. Testing ad content can identify ads that cause backlash and help a brand to choose optimal markets and channels to show ads in.

Survey experiments are a valuable tool for empirically testing ad content. A survey experiment uses data from survey respondents and randomized control trial design to directly estimate the effect of an ad on brand favorability and purchase intent. The methodology is similar to an A/B test but can be run on a small scale before a major launch. The analysis can be used estimate the effect of each on purchase consideration and brand favorability, identify audiences for which an ad is effective, and catch ads that cause backlash. These results of a survey experiment make marketing budgets more efficient, and avoid public relations disasters.

Patrick Miller illustrates the importance of empirically testing ad content before an ad airs through experiments that tested highly controversial ads from Nike and Dove. Out of the hundreds of ads Civis tests each year, the company found about 11% of ads cause backlash. Further, choosing the best ad from several candidates results in a ~13% improvement in key metrics, and up to a 37% improvement for demographic subgroups.

Topics include:

  • Setting up a survey experiment and collecting data using online panel providers
  • Statistical and ML approaches for analyzing data with heterogeneous treatment effects to choose the best markets and channels
  • Weighting the survey so that online sample is representative of the population
  • Presenting the results from modeling to decision makers
Photo of Patrick Miller

Patrick Miller

Civis Analytics

Patrick Miller is a data scientist at Civis Analytics specializing in survey data analysis, causal inference, and production R. Patrick holds a PhD in quantitative psychology, where he studied the applications of machine learning to analysis psychological and behavioral data. You can usually find him at his desk drinking tea and listening to Sufjan Stevens.