Brought to you by NumFOCUS Foundation and O’Reilly Media
The official Jupyter Conference
Aug 21-22, 2018: Training
Aug 22-24, 2018: Tutorials & Conference
New York, NY

Building an Enterprise/Cloud Analytics Platform with Jupyter Enterprise Gateway

Moderated by: Kevin Bates

Notebook kernel processes are the heart and soul of the Notebook application and are responsible for the submission of potentially extreme and resource intensive operations against vast amounts of data, particularly in big data analytics. Running these resource-consuming processes on the same node results in a crippled server struggling to meet the needs of subsequent kernel creation requests. This is hardly a recipe for success within an enterprise servicing dozens or even hundreds of data scientists simultaneously striving to unlock the secrets within their data sets.

With Jupyter Enterprise Gateway, enterprises are able to distribute kernels across the compute cluster, consisting of different capabilities (e.g., GPUs, Cores, Memory, etc.) and leveraging the resource allocation semantics of the underlying resource managers. This is accomplished via a pluggable framework that enables support for additional resource managers allowing your enterprise to leverage this functionality.

Jupyter Enterprise Gateway currently supports for Hadoop YARN, IBM Spectrum Conductor, and Kubernetes resource managers with contributions welcome for others.

Comments on this page are now closed.

Comments

Picture of Kevin Bates
Kevin Bates | SR. SOFTWARE ENGINEER
08/27/2018 9:36am EDT

Please note that these “slides” are for the poster I presented at Wednesday’s poster session. There was no associated breakout session in case you’re searching for that in the conference sessions.

Thank you to all that stopped by and asked questions and shared your scenarios. I hope to hear from many of you later.

Best regards,

Kevin.