How to Install Software and Packages

Using Bash and language cells to install packages and modify the filesystem in a Nextjournal runtime, generating reusable environments.

1.
Standalone Bash Runtimes

Each Nextjournal code cell lives in a runtime, and each runtime has an environment, which is a Docker container with its own filesystem. The first time you insert a Bash cell it will come with a new Bash runtime using the default Minimal Bash environment. For more information on working with Bash in Nextjournal, please see Using Bash.

The underlying Linux distribution is Ubuntu 18.04 LTS, so you can install packages from files with dpkg, or fetch them from repositories with apt-get; however, the stock Nextjournal environments do not have package lists stored, so an apt-get update is required first.

apt-get -qq update
apt-get install fortune cowsay

/usr/games/fortune | /usr/games/cowsay

You can also compile and install from source after getting the right tools and dependencies.

apt-get install -y build-essential git libcurl4-openssl-dev

By cloning a repo and doing a quick compilation, you can check on the values of some *coins.

git clone --recurse-submodules https://github.com/Olavhaasie/coinget
cd coinget
make
make install

coinget list | head

2.
Bash in Other Languages

What if you also need to install packages in R or Julia, where it's more convenient to use the interpreter? The best way to do this is to put a Bash cell into another language's runtime.

This can be accomplished using Change Runtime under a code cell's ··· menu. Under this menu you can create a new runtime based on another environment, or move the cell into an existing environment.

Accessing the Change Runtime menu.

While this makes it possible to do many tasks in a single runtime, it is best to do all installs (from Bash or otherwise) in a separate runtime, and export that to a new environment, ensuring that its versions of programs and packages will be preserved for future reproducibility, even through a Remix. This also prevents having to download and install your tools every time you run, and ensures that the newly installed packages will operate as expected, since it restarts the interpreter.

Here is a Bash cell in an R runtime, used to install libhdf5-dev and its dependencies...

apt-get -qq update
apt-get install -o=Dpkg::Use-Pty=0 -y --no-install-recommends libhdf5-dev

...and that enables build and installation of the h5 package in R.

install.packages("h5")

Any runtime in the article can use the saved environment, and once the article is published the exported environment will also be available for other articles to use via transclusion.

3.
Environment Variables

All of the configuration files can be modified, so you are free to add to ld.so.conf and the like. However, if you need to modify the shell environment variables, the best way to do that is via the runtime's configuration panel.

Here you can see and modify existing environment variables by clicking Show Inherited (after running a cell to load the environment), and add new variables with Add Variable. The runtime will start up with these values defined, and they will all be carried along if the runtime's state is exported as a new environment.

4.
Installing Python Packages

Nextjournal's default Python 2 and 3 environments have a number of basic data and plotting packages installed, including numpy, matplotlib, and plotly:

pip freeze

When additional Python packages are required, you can install them in multiple ways. The easiest is to use conda, which will attempt to install all packages and dependencies in a consistent manner, including system packages and libraries. The Anaconda Cloud has a searchable database of packages and channels—by default conda will select only from the anaconda channel.

conda install descartes pysal

Additional channels can be added with the -c flag.

conda install -c conda-forge cartopy

For packages and versions unavailable via conda, or for installing packages distributed as 'wheel files', pip is available. For any packages that require compilation, gcc is available along with the most common build tools.

pip install quilt mapclassify

You can also use pip to install development versions from git repos.

pip install git+https://github.com/geopandas/geopandas@master

Finally, if a package has a setup.py, you can download and install with that.

git clone https://github.com/ResidentMario/geoplot
cd geoplot
git checkout 328cc9d5f8c02470c6257f2bf2fb2f3c5304530f
python setup.py install

The above cells have created a working installation for running Python geoplot, and the runtime's state has been saved as a new environment. This can now be used to display an example from the geoplot gallery, comparing the populations of large U.S. cities:

quilt install ResidentMario/geoplot_data
2.6s
Language:Python
# Load the data (uses the `quilt` package).
import geopandas as gpd
from quilt.data.ResidentMario import geoplot_data

continental_cities = gpd.read_file(
  geoplot_data.usa_cities()).query('POP_2010 > 100000')
continental_usa = gpd.read_file(geoplot_data.contiguous_usa())

# Plot the figure.
import geoplot as gplt
import geoplot.crs as gcrs
import matplotlib.pyplot as plt

poly_kwargs = { 'linewidth': 0.5, 'edgecolor': 'gray', 'zorder': -1 }
point_kwargs = { 'linewidth': 0.5, 'edgecolor': 'black', 'alpha': 1 }
legend_kwargs = { 'bbox_to_anchor': (0.9, 0.9), 'frameon': False }

ax = gplt.polyplot(continental_usa,
                   projection=gcrs.AlbersEqualArea(central_longitude=-98, 
                                                   central_latitude=39.5),
                   **poly_kwargs)

gplt.pointplot(continental_cities, projection=gcrs.AlbersEqualArea(), ax=ax,
               scale='POP_2010', limits=(1, 80),
               hue='POP_2010', cmap='Blues',
               legend=True, legend_var='scale',
               legend_values=[8000000, 6000000, 4000000, 2000000, 100000],
               legend_labels=['8 million', '6 million', '4 million', 
                              '2 million', '100 thousand'],
               legend_kwargs=legend_kwargs,
               **point_kwargs)

plt.title("Large cities in the contiguous United States, 2010")
plt.savefig("/results/map.svg", bbox_inches='tight', pad_inches=0.1)

5.
Installing Clojure Packages

The default Clojure environment provides Clojure v1.10.0-beta8. Other versions, and any required libraries or packages, can be installed using a deps.edn file. In order to ensure loading when the runtime boots, this file must be created by inserting a Code Listing, setting the type (in the lower right) to Extensible Data Notation, and assigning the deps.edn filename using the ··· menu:

{ :deps { com.hypirion/clj-xchart {:mvn/version "0.2.0"} 
          org.clojure/clojure { :mvn/version "1.10.0-beta8" }}}
deps.edn
Extensible Data Notation

The file can then be mounted into the Clojure runtime's filesystem at boot time using the runtime config menu's Mounts section.

Note that booting will take an extended length of time because of the installation, so it is advisable to save an environment with all of the packages installed. Any required system packages can also be installed via a Bash cell.

apt-get -qq update
apt-get install --no-install-recommends \
  fontconfig libxext6 libxrender1 libxtst6 libxi6

The installation above has been saved as a Clojure xChart environment. The deps.edn file is saved along with all other filesystem changes, making using the environment seamless. A simple example makes use of a GPU to render:

(require '[com.hypirion.clj-xchart :as c])

(def chart
  (c/category-chart
    {"Bananas" {"Mon" 6, "Tue" 2, "Fri" 3, "Wed" 1, "Thur" 3}
     "Apples" {"Tue" 3, "Wed" 5, "Fri" 1, "Mon" 1}
     "Pears" {"Thur" 1, "Mon" 3, "Fri" 4, "Wed" 1}}
  {:title "Weekly Fruit Sales"
   :width 640
   :height 500
   :stacked? true
   :x-axis {:order ["Mon" "Tue" "Wed" "Thur" "Fri"]}}))

(c/spit chart "/results/out.png")

6.
Installing Julia Packages

The Julia 1.0 and 0.7 default environments include a wide variety of file-handling and data processing packages, as well as the Plots and Makie graphical frameworks.

inst = Pkg.installed()
[println("$key v$(inst[key])") for key in keys(inst)];

Installation of additional packages is done via the the standard Julia Pkg interface: update will upgrade all packages with newer versions available, and add will install new packages. Running precompile before saving an environment will speed up package loading in the future.

pkg"add LightGraphs GraphPlot"
pkg"precompile"

The saved Julia GraphPlot environment can be used to display the 'Karate Graph':

using LightGraphs, GraphPlot, Colors

# load a famous graph
g = graphfamous("karate")
nodelabel = collect(1:nv(g))
# node membership
membership = 
  [1,1,1,1,1,1,1,1,2,1,1,1,1,1,2,2,1,1,2,1,2,1,2,2,2,2,2,2,2,2,2,2,2,2]
nodecolor = [colorant"lightseagreen", colorant"orange"]
# membership color
nodefillc = nodecolor[membership]
gplot(g, nodefillc=nodefillc, nodelabel=nodelabel)

7.
Installing R Packages

The R environment includes the tidyverse, and the plotly and ggplot2 graphical packages.

inst = installed.packages()[,c("Package","Version")]
inst[order(tolower(inst[,"Package"])),]

As always, a Bash cell can install dependencies.

apt-get -qq update
apt-get install libudunits2-dev
apt-get clean

Installation of additional R packages is done via the the standard R function install.packages(), or the devtools function install_github(). Build tools, including gcc, are included for package compilation.

install.packages(c("ggraph","igraph"))
devtools::install_github("sjmgarnier/viridis")

An example of circle packing using the created R *graph environment:

# Libraries
library(ggraph)
library(igraph)
library(tidyverse)
library(viridis)
 
# We need a data frame giving a hierarchical structure. Let's consider the flare dataset:
edges=flare$edges
vertices = flare$vertices
mygraph <- graph_from_data_frame( edges, vertices=vertices )
 
ggraph(mygraph, layout = 'circlepack', weight="size") + 
  geom_node_circle(aes(fill = as.factor(depth), color = as.factor(depth) )) +
  scale_fill_manual(values=c("0" = "white", "1" = "white", "2" = magma(4)[2], "3" = magma(4)[3], "4"=magma(4)[4])) +
  scale_color_manual( values=c("0" = "white", "1" = "white", "2" = "black", "3" = "black", "4"="black") ) +
  theme_void() + 
  theme(legend.position="FALSE")