Brought to you by NumFOCUS Foundation and O’Reilly Media Inc.
The official Jupyter Conference
August 22-23, 2017: Training
August 23-25, 2017: Tutorials & Conference
New York, NY

Collaboration and automated operation as literate computing for reproducible infrastructure

Y M (National Institute of Informatics)
2:40pm–3:20pm Friday, August 25, 2017
Usage and application
Location: Sutton Center/Sutton South Level: Beginner

Who is this presentation for?

  • Data scientists

Prerequisite knowledge

  • Basic knowledge of the Jupyter Notebook and infrastructure operation

What you'll learn

  • Explore literate computing practices and methods for combining Jupyter with Ansible for infrastructure operation

Description

We have been applying the Jupyter Notebook to operational engineering for years to meet various research and educational customers’ needs for cloud infrastructure, but Jupyter is useful for DevOps too. It enables collaboration between experts and novices to accumulate infrastructure knowledge, while automation via notebooks enhances traceability and reproducibility. Yoshi Nobu Masatani shows how to combine Jupyter with Ansible for reproducible infrastructure and explores knowledge, workflow, and customer support as literate computing practices.

Nobu shares best practices and use cases for utilizing Jupyter for collaborative communication about design and process and managing reproducible infrastructure throughout its lifecycle. The National Institute of Informatics has centralized the Jupyter Notebook in its cloud operations. Both ad hoc and recurring operations are conducted under Jupyter: every daily ad hoc operation is recorded as a notebook for traceability and reproducibility, and elaborated workflows are shared and standardized as reproducible notebooks and automate operations using Ansible.

For reproducibility, it is as important to be able to share information and processes with others as it is to actually automate complex operations for infrastructures. Jupyter enables users to describe automated operations as live code and share predicted and reproducible outcomes with others. Its narrative stories allow collaborative communication between experts and novices in order to accumulate infrastructure knowledge within an operation team. Moreover, it enables an efficiently shared understanding of how an infrastructure is usable and actually works between tech ops and nontech users. Narrative stories also help communicate with users about delivered services and customized conditions considering reproducibility.

NII’s practices cover the entire infrastructure lifecycle, from initial design and deployment to daily operations and maintenance, rebuilding for new target capacity and throughput, trouble shooting, and customer support and tuned customization. There are available DevOps tools for these tasks, but each automation is within a silo—just a portion of the lifecycle. Enter literate computing, which augments collaboration and makes automation beyond silos possible.

Nobu shares NII’s literate computing practices for reproducible infrastructure. Topics include:

  • Why NII centralizes the Jupyter Notebook and why automation in silos is not enough
  • How the notebook operations are consistently associated to the infrastructure lifecycle by combining Jupyter with Ansible
  • Collaborative communication about design and process and how to organize and refactor Jupyter notebooks and Ansible playbooks in order to have reproducible, reusable, and transferable operations throughout the lifecycle
  • Traceability and reproducibility through narrative stories’ two aspects: Retrospective traceable evidence and prospective reusable guidance
  • How to share common understanding about an infrastructure with customers: How it is usable and what is reproducible
  • Example Jupyter notebooks and Ansible playbooks for reproducible infrastructure operations

For more information, check out a short video, Literate Computing for Reproducible Infrastructure, exploring these practices. You can find a Hadoop example on GitHub, which includes a set of notebooks and Ansible playbooks that cover several phases of a Hadoop cluster’s lifecycle, the initial design (accommodation) and automated deployment, operational alignment, and postdeployment expansion and decommission. Notebook extensions for robust infrastructure operations can be found here.

Photo of Y M

Y M

National Institute of Informatics

Yoshi Nobu Masatani is a project researcher at the National Institute of Informatics, an inter-university research institute for information and systems, where he is responsible for the design and operation of the academic cloud within NII. These years his team utilizes Jupyter for DevOps on NII’s cloud as Literate Computing practices. He has a broad range of experience with OSS-based infrastructure deployments and operations with high-availability systems and big data clusters. Previously, Nobu was a senior specialist and manager of OSS professional services within a large IT company.