Whether a company is seeing rapid growth or has an existing large customer base, the performance of its software is crucial and can be impacted by a range of variables. These variables include how a company delivers applications to customers, how host machines run the software, and everything in between. Performance debugging is a crucial part of ensuring code is production ready, particularly as a company and its products grow. Debugging bottlenecks that prevent existing software from performing optimally can open up a business’s system to scale and handle more usage. However, most of the battle in the debugging process is actually identifying the bottlenecks rather than fixing them. Skills such as tracing, monitoring, and profiling are invaluable in identifying these bottlenecks.
Christian Grabowski shares his experience debugging bottlenecks in distributed systems, at both a macro (metrics, distributed tracing) and a micro (user space and kernel space profiling) level, focusing particularly on tuning REST API services to handle databases that had doubled in size in a matter of a day and taming a resource-hungry, high-throughput metrics ingestion service.
In the macro view, the goal is to identify the bottleneck(s) of a distributed system. Which service is preventing higher throughput? Which service is adding latency? Which service is using all of the resources? Thankfully, there are many available tools to pinpoint the answer to these questions, such as operational metrics and distributed tracing. The micro view, on the other hand, examines where bottlenecks exist in the service itself. This can involve blocks of code, the right balance of resources, or the configuration of the service or machine. Recent technology is emerging to help identify these issues, such as dynamic tracing with things like eBPF.
Join Christian to learn how to overcome and solve these bottlenecks, making software scale and perform substantially better.
Christian Grabowski is a backend software engineer at next-gen DNS and traffic management company NS1. Christian has worn many engineering hats over the course of his career and has worked on a variety of software but loves getting into the nitty-gritty low-level code the most. When he is not developing fast, intelligent DNS services, he’s rather active in the open source community, contributing to projects such as gobpf, bcc, and Kubernetes.
©2018, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org