Presented By
O’Reilly + Intel AI
Put AI to Work
April 15-18, 2019
New York, NY
Discover opportunities for applied AI
Organizations that successfully apply AI innovate and compete more effectively. How is AI transforming your business?
Be a part of the program—apply to speak by October 16.

Massively Parallel Hyperparameter Tuning

Ameet Talwalkar (Carnegie Mellon University and Determined AI)
1:00pm1:40pm Wednesday, April 17, 2019
Machine Learning, Models and Methods
Location: Grand Ballroom West
Secondary topics:  Deep Learning and Machine Learning tools, Models and Methods

Who is this presentation for?

Machine Learning Engineers, Data Scientists



Prerequisite knowledge

Attendees should be familiar with the basics of Machine Learning and Deep Learning.

What you'll learn

There are three main takeaways from this presentation: (1) Hyperparameter tuning is important yet very expensive; (2) We can leverage parallelism to drastically reduce wall-clock time; (3) Traditional hyperparameter search methods are competitive with neural architecture search, while also being simpler / more efficient.


Modern learning models are characterized by large hyperparameter spaces. In order to adequately explore these large spaces, we must evaluate a large number of configurations, typically orders of magnitude more configurations than available parallel workers. Given the growing costs of model training, we would ideally like to perform this search in roughly the same wall-clock time needed to train a single model. In this work, we tackle this challenge by introducing ASHA, a simple and robust hyperparameter tuning algorithm with solid theoretical underpinnings that exploits parallelism and aggressive early-stopping. Our extensive empirical results show that ASHA slightly outperforms Fabolas and Population Based Tuning, state-of-the hyperparameter tuning methods; scales linearly with the number of workers in distributed settings; converges to a high quality configuration in half the time taken by Vizier (Google’s internal hyperparameter tuning service) in an experiment with 500 workers; and matches the performance of complex neural architecture search methods in under 2x the time to train a single model.

Photo of Ameet Talwalkar

Ameet Talwalkar

Carnegie Mellon University and Determined AI

Ameet Talwalkar is co-founder and chief scientist at Determined AI and an assistant professor in the Machine Learning Department at Carnegie Mellon University. His primary interests are in the field of statistical machine learning, including problems at the intersection of systems and learning. He helped to create the SysML conference, led the initial development of the MLlib project in Apache Spark, is the coauthor of the graduate-level textbook Foundations of Machine Learning (MIT Press), and teaches an award-winning MOOC called Distributed Machine Learning with Apache Spark (edX).

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)