Skip to main content

Achieving Rapid Response Times in Large Online Services

Jeff Dean (Google)
Mission City Ballroom
Average rating: ****.
(4.31, 55 ratings)

Today’s large-scale web services provide rapid responses to interactive requests by applying large amounts of computational resources to massive datasets. They typically operate in warehouse-sized datacenters and run on clusters of machines that are shared across many kinds of interactive and batch jobs. As these systems distribute work to ever larger numbers of machines and sub-systems in order to provide interactive response times, it becomes increasingly difficult to tightly control latency variability across these machines, and often the 95%ile and 99%ile response times suffer in an effort to improve average response times. As systems scale up, simply stamping out all sources of variability does not work. Just as fault-tolerant techniques needed to be developed when guaranteeing fault-free operation by design became unfeasible, techniques that deliver predictably low service-level latency in the presence of highly-variable individual components are increasingly important at larger scales.

In this talk, I’ll describe a collection of techniques and practices lowering response times in large distributed systems whose components run on shared clusters of machines, where pieces of these systems are subject to interference by other tasks, and where unpredictable latency hiccups are the norm, not the exception. Some of the techniques adapt to trends observed over periods of a few minutes, making them effective at dealing with longer-lived interference or resource contention. Others react to latency anomalies within a few milliseconds, making them suitable for mitigating variability within the context of a single interactive request. I’ll discuss examples of how these techniques are used in various pieces of Google’s systems infrastructure and in various higher-level online services.

This talk presents joint work with Luiz Barroso and a number of other colleagues at Google.

Photo of Jeff Dean

Jeff Dean

Google

Jeff Dean is a Google senior fellow in Google’s Research Group, where he cofounded and leads the Google Brain team, Google’s deep learning and artificial intelligence research team. He and his collaborators are working on systems for speech recognition, computer vision, language understanding, and various other machine learning tasks. During his time at Google, Jeff has codesigned and implemented many generations of Google’s crawling, indexing, and query serving systems, major pieces of Google’s initial advertising and AdSense for content systems, and Google’s distributed computing infrastructure, including the MapReduce, BigTable and Spanner systems, protocol buffers, LevelDB, systems infrastructure for statistical machine translation, and a variety of internal and external libraries and developer tools. Jeff is a fellow of the ACM and the AAAS, a member of the US National Academy of Engineering, and a recipient of the ACM-Infosys Foundation Award in the Computing Sciences. He holds a PhD in computer science from the University of Washington, where he worked with Craig Chambers on whole-program optimization techniques for object-oriented languages, and a BS in computer science and economics from the University of Minnesota.