Project Jupyter aims at providing a consistent set of tools for interactive computing workflows across multiple programming languages. Jupyter projects are popular at all stages of a research project from the exploration phase to the communication of results and teaching.
The flagship project of Jupyter, the Notebook, and its modernized version, JupyterLab are web applications allowing the creation of documents including prose, executable code, and interactive visualizations.
The kernel is the part of the backend responsible for executing code written by the user in the web application. For example, in the case of a Python notebook, execution of the code is typically handled by ipykernel, the reference implementation.
Initially, the IPython project included everything from the notebook web application to the kernel and server implementation. Later on, the language-agnostic parts (the notebook format, messaging protocol, qtconsole, notebook web application) were split into separate projects with a clearer scope, together forming Project Jupyter.
Splitting IPython into multiple packages was a good decision in that it brought a clear separation of concerns between the projects. The ipykernel project was elected as the reference implementation of the Jupyter kernel protocol.
This came at a cost since ipykernel brought all the history and technical debt of IPython with it. Besides, Python is not a natural language for a base implementation. As a result, R and Julia kernels don’t use the reference implementation but use their own implementation of the protocol.
Furthermore, the standard implementation of the Jupyter Interactive Widgets lies in the Python package ipywidgets. That means that once you, as a kernel developer, implemented the Jupyter kernel protocol, you would still need to make your own backend for the interactive widgets if you want widgets support (and you should want it, honestly…).
To prevent this useless duplication of effort, which harms sustainability, we set ourselves to implement a solid reusable implementation of the Jupyter kernel protocol, xeus.
Xeus is a C++ implementation of the Jupyter kernel protocol. It is not a kernel by itself but a library that helps kernel authoring. Xeus is very helpful when developing a kernel for a language that has a C or a C++API (Python, Lua, SQL, etc.). It takes the cumbersome task of implementing the Jupyter messaging protocol for you so you just can focus on the core interpreter tasks: executing code, inspecting, completing, etc.
C++ is a good choice for a standard implementation of the protocol, it is a common denominator of most of the languages out there, it has a massive developer community and is widely adopted in the industry for performance middleware applications.
Xeus-cling is a xeus-based C++ kernel for the Jupyter Notebook. It’s very useful for teaching or learning C++. C++ has never been this interactive, thanks to the combined power of Jupyter, xeus, and cling. And even more impressive than executing interpreted C++, it supports interactive widgets, thanks to the xwidgets project.
Just like xeus is a C++ implementation of the Jupyter kernel protocol, xwidgets is a C++ implementation of the Jupyter Interactive Widgets protocol, and it can be used as a base for other implementations.
Xeus-python supports error formatting, rich display, interactive widgets, input requests, code completion, code inspection, etc.
Thanks to xeus and pybind11, it was very straightforward to have a first simple version of a Python kernel written in C++. It took a couple of months to have the features listed above, with a relatively small amount of code (< 3000 lines of C++) and a good coverage of ipykernel’s features.
- xeus-python is a lot lighter than ipykernel and IPython combined, which makes it a lot easier to implement new features on top of it. Our next goal is to augment the protocol to implement a Python debugger in JupyterLab.
- xeus-based kernels are more versatile in that one can overload e.g. the concurrency model. This is something that Kitware’s SlicerJupyter project takes advantage of to integrate with the Qt event loop of their Qt-based desktop application.
This development is sponsored by Bloomberg.