Design the Future
January 19–20, 2016: Training
January 20–22, 2016: Conference
San Francisco, CA

Making sentiment surveys practical

Judd Antin (Airbnb)
4:35pm Thursday, 01/21/2016
The new fundamentals
Location: The Chapel
Average rating: ****.
(4.45, 11 ratings)

While big data analytics has quickly innovated to reflect the A/B testing revolution in product design and development, survey methods have failed to keep up. As a result, signals that companies can track at scale, such as clicks and time-on-site, have become the most important decision-making inputs for both tactical and strategic decisions. Along the way, the sentiments of the people who use our products have been marginalized. The potential for better, more holistic decision-making that leverages both behavioral and sentiment data is largely unrealized.

To design better products, we need to be able to precisely track what our users think, but also understand more about why they think it. One driving force behind the lack of innovation in sentiment data collection is the belief that it isn’t practical or scalable. Judd Antin presents tested strategies that are both. These strategies build from several key principles:

  • Organizations often feel the need to adopt one survey approach and try to adapt it for every decision-making situation. This rarely works. Judd presents a typology of decisions and strategies which survey data can inform and relate it to specific ideas for developing relevant survey programs.
  • Judd makes an argument for composite metrics that can roll up into simple metrics that are easy to track but also decompose to help diagnose observed movement in the metric.
  • Judd suggests a bottom-up approach to developing surveys, which starts with the words that users themselves use when they describe their feelings about the product. This method moves from qualitative inquiry to pilot scale development to statistical evaluation in order to hone in on the right questions.
Photo of Judd Antin

Judd Antin


Judd Antin leads the Experience Research team at Airbnb. He and his team embed themselves in product teams to provide quantitative and qualitative research at every stage of the product cycle. Prior to joining Airbnb in May 2015, Judd was a research manager at Facebook and a research scientist at Yahoo! Labs. Judd is a social psychologist who holds a PhD from the I School at UC Berkeley. In 2011, Judd was named one of MIT Technology Review’s “Top 35 Innovators Under 35.”