Analytics data scientists at LinkedIn are responsible for creating metrics to evaluate product success. They’re very good at increasing any metric the company decides to optimize, but following a single metric blindly can lead to problems. For instance, as LinkedIn encouraged members to join conversations, it found itself in danger of creating a “rich get richer” economy in which a few creators got an increasing share of all feedback. Highly skewed distribution of feedback occurs naturally in any system that distributes content virally, but that doesn’t mean it’s good for creators. As a result, LinkedIn implemented changes to balance out this increasing skew and added creator-focused metrics to monitor how well the ecosystem is distributing feedback to creators, which gives the company a more balanced view of the ecosystem.
Drawing on this example, Bonnie Barrilleaux explains why you must regularly reevaluate metrics to avoid perverse incentives—situations where efforts to increase the metric cause unintended negative side effects. Along the way, Bonnie explores a few illustrative historical examples to demonstrate that this problem has been common throughout human endeavors:
These examples remind us that metrics are always an imperfect proxy for reality. In the end, metrics are just a tool for humans to use. It’s important to ensure that the metrics incentivize teams to create real value for users, and it’s critical to regularly evaluate whether the metrics are still effectively driving you in the direction you really want to head. Bonnie shares key defensive strategies to help you do so, by:
Bonnie Barrilleaux is a staff data scientist in analytics at LinkedIn, primarily focused on communities and the content ecosystem. She uses data to guide product strategy, performs experiments to understand the ecosystem, and creates metrics to evaluate product performance. Previously, she completed a postdoctoral fellowship in genomics at the University of California, Davis, studying the function of the MYC gene in cancer and stem cells. Bonnie has published peer-reviewed works including 11 journal articles, a book chapter, and a video article and has been awarded multiple grants to create interactive art. She holds a PhD in chemical engineering from Tulane University.
For exhibition and sponsorship opportunities, email strataconf@oreilly.com
For information on trade opportunities with O'Reilly conferences, email partners@oreilly.com
View a complete list of Strata Data Conference contacts
©2018, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • confreg@oreilly.com