Build & maintain complex distributed systems
October 1–2, 2017: Training
October 2–4, 2017: Tutorials & Conference
New York, NY

Instrumenting systems for arbitrary observability

Baron Schwartz (VividCortex)
2:25pm3:05pm Wednesday, October 4, 2017

Who is this presentation for?

  • Engineers, operators, and developers

Prerequisite knowledge

  • Experience building and running programs in any language

What you'll learn

  • Explore the most useful things to know about observability in systems in production


Observability (or lack thereof), like testability and maintainability, is a fundamental property of systems. But what does observable code look like? What instrumentation creates systems that are observable later in arbitrary ways, in circumstances you can’t foresee? And how can you make your systems observable?

Facing this question at coding time, many programmers try to guess at what’ll be needed later. You can see evidence of this in systems like databases, which provide a lot of “status counters” and the like, most of which were probably included because some developer thought they might be useful. As a result, most end up being vanity metrics. Real production problems never seem to be measured by the things developers thought to include.

When you run most systems, the least common denominator for figuring out how it’s working is usually its log, and the log is usually a deluge of useless “I got here” kinds of signals. So what’s a developer to do? Fortunately, experience operating complex systems suggests that there is a universal set of best practices. Baron Schwartz outlines the most useful things to know about observability in systems in production, helping you instrument your systems in ways that support troubleshooting and operate them in production under failure modes you can’t imagine when you’re writing the code.

Photo of Baron Schwartz

Baron Schwartz


Baron Schwartz is the founder and CTO of VividCortex, the best way to see what your production database servers are doing. Baron has written a lot of open source software and several books, including High Performance MySQL. He’s focused his career on learning and teaching about performance and observability of systems generally, including the view that teams are systems and culture influences their performance, and databases specifically.