Notebook kernel processes are the heart and soul of the Notebook application and are responsible for the submission of potentially extreme and resource intensive operations against vast amounts of data, particularly in big data analytics. Running these resource-consuming processes on the same node results in a crippled server struggling to meet the needs of subsequent kernel creation requests. This is hardly a recipe for success within an enterprise servicing dozens or even hundreds of data scientists simultaneously striving to unlock the secrets within their data sets.
With Jupyter Enterprise Gateway, enterprises are able to distribute kernels across the compute cluster, consisting of different capabilities (e.g., GPUs, Cores, Memory, etc.) and leveraging the resource allocation semantics of the underlying resource managers. This is accomplished via a pluggable framework that enables support for additional resource managers allowing your enterprise to leverage this functionality.
Jupyter Enterprise Gateway currently supports for Hadoop YARN, IBM Spectrum Conductor, and Kubernetes resource managers with contributions welcome for others.
Comments on this page are now closed.
For exhibition and sponsorship opportunities, email email@example.com
For information on trade opportunities with JupyterCon, email firstname.lastname@example.org
View a complete list of JupyterCon contacts
©2018, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com
Please note that these “slides” are for the poster I presented at Wednesday’s poster session. There was no associated breakout session in case you’re searching for that in the conference sessions.
Thank you to all that stopped by and asked questions and shared your scenarios. I hope to hear from many of you later.