Real-time collaboration with Jupyter notebooks using CoCalc

Description

William Stein launched the Sage Notebook in 2006, initially motivated by a talk at Sage Days 1 about a GUI for IPython. CoCalc is a continuation of this project using more modern technology. CoCalc includes a distinct full stack implementation of both the frontend and backend parts of Jupyter, built from scratch using React, Node.js, and Kubernetes. Compatibility with Jupyter—both the published wire protocol and the general feel of UI—is a fundamental design goal.

The core goal of CoCalc is to provide easy, safe, and beginner-friendly access to all open source mathematics and data science software, including SageMath, LaTeX, R, Anaconda, and a large number of Jupyter kernels. The motivation for fully supporting multiple simultaneous people editing Jupyter notebooks is that it makes it much easier for teachers to support their students, for students to support each other, and for the people who run CoCalc to support users. The same technology used to implement real-time collaboration also provides a complete history of all modifications of the notebook. With a granularity of about two seconds, people can follow exactly how a document evolved over time. Hence, there is no longer the fear of “messing everything up,” because it is easy to go back in time. Additionally, professors can follow exactly how students solved a given problem. People can collaborate without having to learn Git or fiddle around with nbdiff.

CoCalc is primarily a whole product designed to make life easier for teachers who want to use open source data science software in teaching beginners with minimal extra effort. College professors are very busy, so CoCalc is built as a single centralized service that is hosted in Google’s cloud and run by SageMath, Inc., rather than something professors or staff have to install and run themselves. Moreover, a huge amount of software is preinstalled in the standard image that CoCalc projects use; this includes everything that anybody has ever requested since 2013 that the team could figure out how to install.

William explains how CoCalc relates to Project Jupyter and shares how he implemented real-time collaborative editing of Jupyter notebooks in CoCalc. CoCalc tackles many of the same problems as JupyterLab and JupyterHub but with very different design constraints, motivations, and results. For example, real-time collaboration has been a core feature of CoCalc since it launched in 2013, whereas classical Jupyter does not have real-time collaboration support; on the other hand, drag-and-drop and a flexible plugin architecture are central features of JupyterLab but are mostly absent in CoCalc. Another subtle difference is that in CoCalc if you close your browser while a computation is running, then open it later, all your output will be there, whereas classical Jupyter discards that output (though it displays output more quickly). Implementing real-time collaboration in a web application involves many choices and trade-offs. CoCalc has had real-time collaboration support for five years. It is optimized for what users need, the sort of documents they edit, and the choices made for how to store data, both in the long and short term.