Matthew Seal explores notebooks as a unifying mechanism for developing, tracking, and debugging small units of work that need to be managed and scheduled and demonstrates how papermill, an nteract tool, can be used to execute notebooks as immutable pieces of code. Matthew explains how this tooling makes notebooks a solid choice for templates in scheduled processes and shares how Netflix is using this pattern to colocate tasks written by users ranging from nonprogrammers to professional system maintainers.
This technology choice and its application stems from a desire to help solve a fundamental problem found in many large code ecosystems. As development environments grow and expand to include more tools, more languages, and more flexibility, it often becomes increasingly difficult to maintain a few simple interfaces that can take advantage of these systems. The task of executing a piece of code within such an ecosystem changes from a single point of entry to many dissimilar and constrained entry points. Learning each of these can be tedious and is a major barrier to entry for new users.
The goal of showing notebooks as traceable units that can be referenced to point-in-time execution is to help alleviate this pain. Matthew details how Netflix targets similar working environments between local development and scheduled tasks without leaving a Jupyter client. When an error occurs in scheduled work, you can debug the problem in the same way you’d debug a local problem. You’ll see some examples of this pattern when pulling failed notebooks from a scheduler and fixing the problems without needing to interact with the intervening technologies.
Matthew Seal is a senior software engineer at Netflix, where he works on scaling data platform solutions. Based in the Bay Area of California, Matthew attended Stanford University for undergraduate and graduate school. He stayed in the area, working at startups and spending a long stretch of time working at OpenGov.
©2018, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org