Testing ad content with survey experiments

Description

Brands spend lots of money on ads but rely on intuition to choose the ad to show to an audience and avoid ads that cause backlash. Data scientists can help marketing departments optimize their decision making by testing the content of ads before they are shown to audiences. Testing ad content can identify ads that cause backlash and help a brand to choose optimal markets and channels to show ads in.

Survey experiments are a valuable tool for empirically testing ad content. A survey experiment uses data from survey respondents and randomized control trial design to directly estimate the effect of an ad on brand favorability and purchase intent. The methodology is similar to an A/B test but can be run on a small scale before a major launch. The analysis can be used estimate the effect of each on purchase consideration and brand favorability, identify audiences for which an ad is effective, and catch ads that cause backlash. These results of a survey experiment make marketing budgets more efficient, and avoid public relations disasters.

Patrick Miller illustrates the importance of empirically testing ad content before an ad airs through experiments that tested highly controversial ads from Nike and Dove. Out of the hundreds of ads Civis tests each year, the company found about 11% of ads cause backlash. Further, choosing the best ad from several candidates results in a ~13% improvement in key metrics, and up to a 37% improvement for demographic subgroups.

Topics include:

• Setting up a survey experiment and collecting data using online panel providers
• Statistical and ML approaches for analyzing data with heterogeneous treatment effects to choose the best markets and channels
• Weighting the survey so that online sample is representative of the population
• Presenting the results from modeling to decision makers