Google’s approach to distributed systems observability

Description

Google has been doing microservices for more than a decade, and the way microservices have progressed at Google mirrors the larger progression of microservices in the industry: a common container format; a way to express complicated systems in terms of containers; the tools to deploy them and services to schedule them; core networking services to support the complicated networking requirements of very large systems with complex dependencies; and observability—collecting diagnostics data to not just recognize and debug production problems but also provide critical signals about usage to Google’s highly adaptive and scalable environment. (A significant core component in Google’s story was the instrumentation of its services and collection of diagnostics data.)

Jaana Burcu Dogan outlines key approaches in instrumenting Google’s services, shares best practices and lessons learned related to patterns, UX, performance, and security, and discusses the company’s recent work to open-source its internal stack. Jaana also offers a demo of gRPC to illustrate how Google autoinstruments libraries and frameworks.

Topics include:

• The case for a common framework
• Core components of an observability framework
• Having instrumentation on by default
• Aggregation of instrumentation data at the node
• White box versus black box strategies
• Z-pages
• The future of instrumentation data