Cultivating production excellence: Taming complex distributed systems

Description

Taming the complex distributed systems you’re responsible for requires changing not just the tools and technical approaches you use, it also requires changing who’s involved in production, how they collaborate, and how you measure success.

Liz Fong-Jones explains practices core to production excellence: giving everyone a stake in production, collaborating to ensure observability, measuring with service-level objectives, and prioritizing improvements using risk analysis. Successful long-term approaches to production ownership and DevOps require cultural change in the form of production excellence. Teams are more sustainable if they have well-defined measurements of reliability, the capability to debug new problems, a culture that fosters spreading knowledge, and a proactive approach to mitigating risk. While tools can play a part in supporting a reliable system, culture and people are the most important investment.

Without a mature observability and collaboration practice, a system will crumble under the weight of technical debt and falter no matter how many people and how much money is ground into the gears of the machine. The leadership of engineering teams must be responsible for creating team structures and technical systems that can sustainably serve user needs and the health of the business.