Perverse incentives in metrics: Inequality in the like economy

Description

Analytics data scientists at LinkedIn are responsible for creating metrics to evaluate product success. They’re very good at increasing any metric the company decides to optimize, but following a single metric blindly can lead to problems. For instance, as LinkedIn encouraged members to join conversations, it found itself in danger of creating a “rich get richer” economy in which a few creators got an increasing share of all feedback. Highly skewed distribution of feedback occurs naturally in any system that distributes content virally, but that doesn’t mean it’s good for creators. As a result, LinkedIn implemented changes to balance out this increasing skew and added creator-focused metrics to monitor how well the ecosystem is distributing feedback to creators, which gives the company a more balanced view of the ecosystem.

Drawing on this example, Bonnie Barrilleaux explains why you must regularly reevaluate metrics to avoid perverse incentives—situations where efforts to increase the metric cause unintended negative side effects. Along the way, Bonnie explores a few illustrative historical examples to demonstrate that this problem has been common throughout human endeavors:

• Rat control in Hanoi: Instead of leading to effective control of the rat population as intended, paying locals by the rat tail actually led to rat farming and a worsening rat population problem.
• “Hoy no circulada” days in Mexico City: Instead of reducing air pollution and boosting public transit ridership as intended, banning certain cars from driving on certain days of the week actually led to increased air pollution, as families bought second cars and kept older higher-polluting cars longer to get around the ban.

These examples remind us that metrics are always an imperfect proxy for reality. In the end, metrics are just a tool for humans to use. It’s important to ensure that the metrics incentivize teams to create real value for users, and it’s critical to regularly evaluate whether the metrics are still effectively driving you in the direction you really want to head. Bonnie shares key defensive strategies to help you do so, by:

• Building a data-informed culture, in which data informs, but humans are always driving
• Creating awareness of the problem so everyone is trained in how to spot it
• Monitoring a balanced suite of metrics to detect and mitigate perverse incentives as they arise
• Reevaluating metrics at regular intervals; if they’re not working, change them