Presented By O'Reilly and Cloudera
Make Data Work
March 13–14, 2017: Training
March 14–16, 2017: Tutorials & Conference
San Jose, CA

Docker on YARN

Daniel Templeton (Cloudera)
1:50pm2:30pm Wednesday, March 15, 2017
Hadoop platform and applications
Location: LL21 E/F Level: Intermediate
Secondary topics:  Architecture
Average rating: ****.
(4.00, 4 ratings)

Who is this presentation for?

  • Hadoop developers and administrators

Prerequisite knowledge

  • A working knowledge of Hadoop (MapReduce or Spark) and Docker

What you'll learn

  • Understand how Docker is supported on YARN
  • Discover plans for integrating Slider into YARN
  • Learn when Docker on YARN is a good solution

Description

Docker makes it easy to bundle an application with its dependencies and provide full isolation, and YARN now supports Docker as an execution engine for submitted applications. Daniel Templeton explains how YARN’s Docker support works, why you’d want to use it, and when you shouldn’t.

Daniel starts with a very short summary of Docker and the use cases that drive its adoption before jumping into how Docker is supported on YARN. Daniel covers container executors and the various options and digs into the Docker support provided by the LinuxContainerExecutor, and he’ll talk about what it doesn’t do yet, where that creates problems, and the work going on in the community to address the gaps. Along the way, Daniel also explores Apache Slider and the Docker capabilities it brings as it’s being merged into YARN and the use cases for when you’d want to use Docker in a Hadoop environment and when there might be a better alternative, covering both traditional applications, like MapReduce jobs, and longer-running services.

You’ll leave with a concrete idea of what’s possible today, what’s coming soon, and when to look to Docker as the solution.

Photo of Daniel Templeton

Daniel Templeton

Cloudera

Daniel Templeton has a long history in high-performance computing, open source communities, and technology evangelism. Today Daniel works on the YARN development team at Cloudera, focused on the resource manager, fair scheduler, and Docker support.