Scalable deep learning

Description

Although deep learning has received much acclaim due to its widespread empirical success, fundamental bottlenecks exist when attempting to develop deep learning applications at scale. One involves exploring the design space of a model family, which typically requires training tens to thousands of models with different hyperparameters. Model training itself is a second major bottleneck, as classical learning algorithms are often infeasible for the petabyte datasets that are fast becoming the norm.

Ameet Talwalkar shares research on addressing these two core scalability bottlenecks. Ameet first offers an overview of Hyperband, a novel algorithm for hyperparameter optimization that is simple, flexible, theoretically sound, and an order of magnitude faster than leading competitors. He then presents work aimed at understanding the underlying landscape of training deep learning models in parallel and distributed environments and introduces Paleo, an analytical performance model that can quickly and accurately model the expected scalability and performance of putative parallel and distributed deep learning systems.