If you can’t measure it, you can’t improve it: How reporting and experimentation fuel product innovation at LinkedIn

Description

Making data-driven decisions through experimentation is an extremely important part of the culture at LinkedIn. It’s deeply ingrained in the company’s development process and has always been a core part of its DNA. LinkedIn tests everything from complete redesigns of its homepage to changes to the backend relevance algorithm to infrastructure upgrades. It’s how the company innovates, grows, and evolves its products to best serve its members. It’s how LinkedIn makes its members happier, its business stronger, and its talent more productive.

Kapil Surlaker and Ya Xu explain why, to meet the company’s needs, LinkedIn built the UMP and XLNT platforms for metrics computation and experimentation, respectively, which have allowed the company to perform measurement and experimentation very efficiently at scale while preserving trust in data.

The Unified Metrics Platform (UMP) is the source of truth for all metrics at LinkedIn. It ensures the definition of a given metric stays the same independent of where the metric is being used. Users are responsible for authoring the metric logic, and the platform is responsible for executing the logic correctly and efficiently and preserving the SLAs. UMP also detects duplicate metric definitions and alerts the metrics’ authors so there is no redundant computation. A decentralized authoring and review process also means that LinkedIn can be agile while still preserving trust in data.

XLNT allows users to quickly quantify the impact of any A/B test in a scientific and controlled manner across LinkedIn.com and its apps, enabling the easy design and deployment of experiments and providing automatic analysis that is crucial in popularizing A/B tests. The platform is generic and extensible, covering almost all domains, including mobile and email. Every day, hundreds of experiments are run and thousands of metrics computed to improve every aspect of LinkedIn.

Initially, daily metric computation was sufficient to meet most needs, but in many cases, this was far from desirable. During rapid product iterations—the norm at LinkedIn—understanding the impact an experiment has on a product metric soon after enabling it is crucial. In 2016, LinkedIn enhanced the platform to support intraday computation so that users could get reports and monitor the impact of experiments on an hourly basis. More recently, the platform has been further enhanced to enable near-real-time computation, and the metrics have been made available everywhere.

Kapil and Ya offer an overview of both platforms, covering the infrastructure that powers them and the rich visualizations that delight LinkedIn’s users. They also share their experiences of running these platforms in production over the past few years at LinkedIn and explain how they have helped shape the LinkedIn product.