Brought to you by NumFOCUS Foundation and O’Reilly Media Inc.
The official Jupyter Conference
August 22-23, 2017: Training
August 23-25, 2017: Tutorials & Conference
New York, NY

All Kernels Go: Building a Rich Environment for Interactive HPC with JupyterHub

Moderated by: Ashwin Trikuta Srinath, Linh Ngo, & Jeff Denton

Who is this presentation for?

Research Support, HPC Support, Faculty, Instructor, Research Assistant

Prerequisite knowledge

* Some familiarity with Linux from an administrative viewpoint (e.g., recognize the terms "permissions" and "root" in the context of Linux). * Basic knowledge about batch-scheduled HPC systems (e.g., recognize the terms "scheduler", "module" and "job" in the context of HPC) * Some knowledge about JupyterHub is beneficial but not required.

What you'll learn

* An example of deploying JupyterHub in HPC * Common problems and gotchas in deploying JupyterHub in HPC * What to provide/support in an HPC JupyterHub Deployment

Description

JupyterHub has been successfully deployed at various high-performance computing (HPC) centers, as a means to enable users to perform large-scale, data-intensive computations in Jupyter Notebook. In this talk, we will provide a brief overview of our deployment of JupyterHub, which allows our users to use Jupyter Notebook with two large-scale computing resources: a medium-sized (2,000 node, 20,000 core) traditional HPC cluster, and a 3.64 PB, 40-node Hadoop cluster. This deployment supports a wide variety of use-cases in research and teaching at our institute. We will present several examples of these and discuss solutions to several practical problems that we encountered in supporting them:

1. Enabling parallel programming in Notebooks: e.g., MPI, GPUs and Spark
2. Environment modules and customizable shell environments in Jupyter Notebooks
3. Custom kernels for Jupyter Notebooks
4. Integrating Singularity containers and Jupyter Notebook

This talk is ideal for anyone interested in, or currently deploying JupyterHub in an HPC environment. Previous knowledge of JupyterHub is not strictly required but would be beneficial.