Put AI to work
June 26-27, 2017: Training
June 27-29, 2017: Tutorials & Conference
New York, NY

Integrating deep learning libraries with Apache Spark

Joseph Bradley (Databricks), Xiangrui Meng (Databricks)
2:35pm3:15pm Thursday, June 29, 2017
Implementing AI
Location: Sutton South/Regent Parlor Level: Intermediate
Secondary topics:  Cloud, Deep Learning

Prerequisite Knowledge

  • Experience with deep learning and Apache Spark (useful but not required)

What you'll learn

  • Learn how to scale deep learning and integrate popular deep learning libraries with Apache Spark


The combination of deep learning with Apache Spark has the potential to make a huge impact. Joseph Bradley and Xiangrui Meng share best practices for integrating popular deep learning libraries with Apache Spark. Rather than comparing deep learning systems or specific optimizations, Joseph and Xiangrui focus on issues that are common to many deep learning frameworks when running on a Spark cluster, such as optimizing cluster setup and data ingest (clusters can be configured to avoid task conflicts on GPUs and to allow using multiple GPUs per worker), configuring the cluster (setting up pipelines for efficient data ingest improves job throughput), and monitoring long-running jobs (interactive monitoring facilitates both the work of configuration and checking the stability of deep learning jobs). Joseph and Xiangrui then demonstrate the techniques using Google’s popular TensorFlow library.

Photo of Joseph Bradley

Joseph Bradley


Joseph Bradley is a software engineer working on machine learning at Databricks. Joseph is an Apache Spark committer and PMC member. Previously, he was a postdoc at UC Berkeley. Joseph holds a PhD in machine learning from Carnegie Mellon University, where he focused on scalable learning for probabilistic graphical models, examining trade-offs between computation, statistical efficiency, and parallelization.

Photo of Xiangrui Meng

Xiangrui Meng


Xiangrui Meng is an Apache Spark PMC member and a software engineer at Databricks. His main interests center around developing and implementing scalable algorithms for scientific applications. Xiangrui has been actively involved in the development and maintenance of Spark MLlib since he joined Databricks. Previously, he worked as an applied research engineer at LinkedIn, where he was the main developer of an offline machine learning framework in Hadoop MapReduce. He holds a PhD from Stanford, where he worked on randomized algorithms for large-scale linear regression problems.