Leverage the Power of Runtimes

In this short tutorial we try to demonstrate the full potential of runtimes in Nextjournal. This is one of the most unique features of Nextjournal compared to other computational notebooks.

Maybe you had the chance to read the documentation of Runtimes and Environments which gives you the technical background. Here, we focus more on the application; we will show you step-by-step how to use runtimes in a more sophisticated way, and what are the best use-cases. You will learn how to create an environment, export it, import it, and how to install libraries and functions. All these features are language-agnostic, but for demonstration purposes we use Python in this tutorial.

The best way to use this tutorial is to open an account (it's free) and then click the Remix button in the menu bar. You can execute the code cells along the way, which will help you understand the concepts. The notebook will also appear in your personal dashboard, where you can find it and start playing around with it. Another option is to just create a new notebook, and execute the commands described in the tutorial performing each single step yourself. This way you will probably learn the most.

Installing libraries

The basic concept of using separate runtimes for installing and importing libraries is the idea that we do not want to re-install libraries over and over again. To avoid this repetitive and time consuming task, we create a new runtime where we install the desired package. By then exporting the environment, we create a Docker image with that package and Nextjournal will push it to a central repository. This means that any other runtime can use this Docker image as base image, where the libraries are already pre-installed. This has another huge advantage, we can make sure that we use the correct version of the libraries and can use Nextjournal's automatic versioning to find out when we used which version.

Let's do it step-by-step. We use one runtime as our research runtime, let's create a code cell in Python.

It is a good idea to give runtimes a meaningful name, so let's change the name of this runtime to research by clicking the gear sign next to the runtime, which can be found in the Table of Contents section of the sidebar.

Now, we would like to use a library that is not yet pre-installed in the standard Python 3 environment. We create another runtime by adding a code cell but this time we choose Bash and we need to change the runtime to Python 3 by either opening the options next to the cell (clicking the “•••” button or by Command/Ctrl + Shift + O) , then selecting Change Runtime.... and subsequently Add New Runtime..., and choose Python 3. As an alternative to doing these steps, clicking on the gear sign of that runtime you can choose the environment and pick Python 3 from there.

The default Bash runtime gets attached to the previous runtime which is research in our case but we want a fully independent runtime. We install xgboost via pip in the Bash runtime (the below code cell is just an image, because we moved the runtime to the Appendix).

We give the runtime a more meaningful name, xgboost in our case. Next, we need to export it, by clicking the gear sign next to the runtime. Scroll all the way down, and you will find the option to export the environment. You will see a warning in orange, below the runtime in the Table of Contents, that asks you to Review & Save the environment.

Doing so will create the Docker image and upload it to the registry. The location and name of this Docker image is also displayed in the runtime settings. It should now look like this:

That means you can use that Docker image from any other notebook, and also import it anywhere else, e.g. your local machine or a production server.

Let's use this environment as the base for our research runtime. We click the gear sign next to the runtime and choose xgboost as the environment. Just start typing xg... and it will auto-complete.

Now the xgboost library is available within the research runtime.

import xgboost

As you can see below we moved the code cell where we installed the xgboost library into the Appendix, which is a good idea to keep our notebook clean.

It is also a good idea to lock the cell where we installed the library to not accidentally re-run the cell again. This allows us as well to choose Run all from the run options without worrying that the installation will be triggered again. Just click the “•••” button next to the cell, or use Command/Ctrl+Shift+O and choose the Lock cell option.

With this paradigm, we can create environments with pre-installed packages that everyone can import and use. If we update a library to a newer version, we must not forget to re-import the latest environment. Since Nextjournal emphasizes sharing and reproducibility, an import will not automatically update in order to keep the state of a notebook unchanged until we actively make that change.

Installing GitHub repositories

In case you need to install from a GitHub repository, and want to make the code available in other notebooks without re-installing the repository over and over again, just create an environment with it.

The first step is to add a repository by choosing GitHub Repository from the menu triggered by the Add Content button (or the “+” button below the block that you are working on). If you do so, you see the following

Next, you need to enter the name of the repository, and then you need to choose which runtime should have access. In our case we want the research runtime to use the repo.

We tick research and apply and restart the runtime.

Next step is to add the automatically mounted repository to the PYTHONPATH or our runtime so we can import and use it.

You see that the demo package is automatically added to the mounts. This changes the runtime you you need to save the changes, you should see:

Review the changes and save them, or save and restart.

Now we can simply use this package in our runtime and import it.

pip install /demo

pip show demo

from demo.mammals import Mammals

Mammals().printMembers()

Exporting the environment

So far we have done two things, use another environment xgboost as base environment for the research environment and installed a GitHub repository in the research environment. It is very useful to export this environment as well so we can use it in other notebooks and use xgboost and the demo library without installing it again.

Click on the gear sign next to the runtime and scroll all the way down.

From there, choose to Export the environment and you will have to Review & Save the changes.

Now you will see that the environment was added to the Nextjournal Docker registry and you can access it from any other notebook, or download the image.

Importing the environment is simple and with the exact same steps as above. In any notebook, just click on the gear sign of the current runtime, click on the Environment dropdown and choose the Import environment... option. You need to go through the profiles, and add the runtime from the notebooks; Leverage the power of runtimes in this case, and then the research environment. Now you can use the xgboost and demo packages right away. Please remember, in case you update e.g. to a new version of xgboost in this notebook you need to import the latest version in the other notebook.

Summary

This short tutorial showed you the powerful concept of runtimes and how to use them to install libraries and packages.

If you also think the world needs more reproducibility and collaboration, Nextjournal is free for private use, so you can sign-up and start using right away.

Was there something we missed in our tutorial? Please feel free to remix, add your work and share it with us and the world!

Read more awesome notebooks created in Nextjournal related to Data Science, Machine Learning, Scientific Publishing here.

If you have questions about us, the pricing, or how we can help you to set up Nextjournal for your team, get in touch!

Happy collaborating ❤️

Appendix

pip install xgboost