JupyterHub Meets CloudyCluster

Description

It is well known that Jupyter Notebook provides a rich environment for interfacing with the resources of one or even multiple computers. JupyterHub provides a pluggable system for authenticating users and placing a user’s Jupyter Notebook server on some computing resource. Some fortunate institutions with HPC resources have paired JupyterHub with their HPC clusters to the benefit of their members, but what about everyone else that doesn’t have access to the same levels of infrastructure and support?

CloudyCluster puts the power in its users’ hands by providing a self-service, responsive interface that enables an individual or organization to:

• Launch pre-configured or customizable clusters
• Invite collaborators to use a cluster, including the ability to use Jupyter
• Customize the software available on a cluster
• Benefit from auto-scaling clusters via Cloudy Cluster Queue (CCQ)
• Cost-effectively support many concurrent users and/or support large-scale computations

We have integrated the JupyterHub community project “batchspawner” with the CCQ meta-scheduler to enable JupyterHub to leverage auto-scaling, batch-scheduled resources. This auto-scaling feature allows a CloudyCluster user to operate a minimal cluster until the user or one of their collaborators submits a job that requires additional resources. Those resources are then instantiated by CCQ and live for the duration it is needed by the requestor. When the CCQ job ends, the resources that are no longer needed are scaled back, significantly reducing costs compared with static, on-premises solutions.

CloudyCluster offers a pay-as-you-go model familiar to those already operating in the cloud.