Put open source to work
July 16–17, 2018: Training & Tutorials
July 18–19, 2018: Conference
Portland, OR

Google’s approach to distributed systems observability

Jaana B. Dogan (Google)
11:50am12:30pm Thursday, July 19, 2018
Distributed computing
Location: Portland 255
Level: Beginner
Average rating: ****.
(4.25, 4 ratings)

Who is this presentation for?

  • Software engineers, site reliability experts, and those working in ops

Prerequisite knowledge

  • A basic understanding of microservices architecture, distributed tracing, and monitoring

What you'll learn

  • Learn how Google builds and maintains an observability framework in a large-scale microservices environment


Google has been doing microservices for more than a decade, and the way microservices have progressed at Google mirrors the larger progression of microservices in the industry: a common container format; a way to express complicated systems in terms of containers; the tools to deploy them and services to schedule them; core networking services to support the complicated networking requirements of very large systems with complex dependencies; and observability—collecting diagnostics data to not just recognize and debug production problems but also provide critical signals about usage to Google’s highly adaptive and scalable environment. (A significant core component in Google’s story was the instrumentation of its services and collection of diagnostics data.)

Jaana Burcu Dogan outlines key approaches in instrumenting Google’s services, shares best practices and lessons learned related to patterns, UX, performance, and security, and discusses the company’s recent work to open-source its internal stack. Jaana also offers a demo of gRPC to illustrate how Google autoinstruments libraries and frameworks.

Topics include:

  • The case for a common framework
  • Core components of an observability framework
  • Having instrumentation on by default
  • Aggregation of instrumentation data at the node
  • White box versus black box strategies
  • Z-pages
  • The future of instrumentation data
Photo of Jaana B. Dogan

Jaana B. Dogan


Jaana B. Dogan is a software engineer at Google, where she works on observability of Go production services. She has a decade of experience building developer platforms and tools.