Design the Future
January 19–20, 2016: Training
January 20–22, 2016: Conference
San Francisco, CA

Measuring hard-to-measure things

1:15pm Thursday, 01/21/2016
The new fundamentals
Location: The Chapel
Average rating: ****.
(4.43, 7 ratings)
Slides:   external link

Early product success is both exciting and terrifying because if the growth path is up and to the right, your audience will broaden, as will your challenges. Newcomers will arrive with experiences less similar to those of your earliest users and may begin using your product in ways you didn’t intend, leaving you puzzled. In GitHub’s case, there was a distinct shift in growth and audience: by 2013, new signups by developers with limited experience using version control and Git far outnumbered those by experienced developers. The cache of a GitHub membership within the developer community brought many people through the door and into a product that is frankly pretty hard to figure out. As a result, more people were failing than succeeding.

All products have blind spots, and no amount of analytics and monitoring can account for hidden variables. The sheer volume of data we have now can make it hard for talented analysts to utilize information, and powerful tools and visualizations are rendered less useful. And even the best systems can gather data on unusual behaviors for a long time without signaling that something is amiss. At GitHub, with regard to engineering challenges, data collection, and analysis, we tend to think in terms of technical instruments. If you want to surface blind spots, you will have to get out from behind the numbers and in front of people. Think of yourself as a human instrument.

GitHub’s Chrissie Brodigan covers three examples of how GitHub uses research to deliver product and connect with customers: how it measures new user uptake with longitudinal survey analysis and failure with exit surveys, how and why we roll out new features on (like pricing studies) as controlled experiments, and how it designs feature prioritization sneak attacks (learning through asking the right question) to learn more about customer culture.

Photo of Chrissie Brodigan

Chrissie Brodigan


Chrissie Brodigan is a user experience researcher and a former historian of airline stewardesses. Chrissie leads user experience research for GitHub. She previously worked at Mozilla and with a number of startups, designing experiences to help onboard new users. Long before she was a user researcher at GitHub, she spent most of her time studying and writing about gender and labor with an emphasis on women’s careers in aviation.