Notebook kernel processes are the heart and soul of the Notebook application and are responsible for the submission of potentially extreme and resource intensive operations against vast amounts of data, particularly in big data analytics. Running these resource-consuming processes on the same node results in a crippled server struggling to meet the needs of subsequent kernel creation requests. This is hardly a recipe for success within an enterprise servicing dozens or even hundreds of data scientists simultaneously striving to unlock the secrets within their data sets.
With Jupyter Enterprise Gateway, enterprises are able to distribute kernels across the compute cluster, consisting of different capabilities (e.g., GPUs, Cores, Memory, etc.) and leveraging the resource allocation semantics of the underlying resource managers. This is accomplished via a pluggable framework that enables support for additional resource managers allowing your enterprise to leverage this functionality.
Jupyter Enterprise Gateway currently supports for Hadoop YARN, IBM Spectrum Conductor, and Kubernetes resource managers with contributions welcome for others.
Comments on this page are now closed.
©2018, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org