Presented By O’Reilly and Cloudera
Make Data Work
September 11, 2018: Training & Tutorials
September 12–13, 2018: Keynotes & Sessions
New York, NY

Deep learning on YARN: Running distributed TensorFlow, MXNet, Caffe, and XGBoost on Hadoop clusters

Wangda Tan (Hortonworks)
1:10pm–1:50pm Thursday, 09/13/2018
Data engineering and architecture
Location: 1A 10 Level: Intermediate
Secondary topics:  Data Platforms, Deep Learning, Model lifecycle management
Average rating: ****.
(4.50, 2 ratings)

Who is this presentation for?

  • Solution engineers, data engineers, infrastructure engineers, and CxOs

Prerequisite knowledge

  • A basic understanding of YARN and deep learning frameworks

What you'll learn

  • Learn how to easily run applications such as TensorFlow, MXNet, Caffe, and XGBoost on YARN


Deep learning is useful for enterprises tasks such as speech recognition, image classification, AI chatbots, and machine translation, just to name a few. In order to train deep learning and machine learning models, you must leverage applications such as TensorFlow, MXNet, Caffe, and XGBoost.

Wangda Tan discusses new features in Apache Hadoop 3.x to better support deep learning workloads, such as first-class GPU support, container-DNS support, scheduling improvements, and more. These improvements make running distributed deep learning and machine learning applications on YARN as simple as running them locally, which allows machine learning engineers to focus on algorithms instead of worrying about the underlying infrastructure. Wangda then demonstrates how to run these applications on YARN.

Photo of Wangda Tan

Wangda Tan


Wangda Tan is a senior member of the technical staff at Hortonworks, where he focuses on the Hadoop YARN resource scheduler. Previously, he worked on integrating OpenMPI with Hadoop at Pivotal and created a large-scale machine learning and matrix computation platform at Alibaba, which now runs on 5,000 clusters and handles some most complex parallel computation tasks at the company.