The Nextjournal Bash Environment

The default environment for standalone Bash cells

The Bash environment is the default for a standalone Bash cell, and also serves as the base for most other images.

Showcase

The environment is based on the ubuntu:18.04 docker image.

lsb_release -d
0.9s
Bash Showcase (Bash)
Bash

These packages are installed in the default Minimal Bash environment.

dpkg -l | tr -s " " | sed -e '1,3d; 5d; s/ /,/g; s/,/ /5g' | \
  sed -e "s/\"/'/g" | awk -F"," '{printf "\"%s\",\"%s\",\"%s\"\n",$2,$3,$5}' \
  > /results/pkgs.csv
echo "Name,Version
  $(gcloud --version | sed 's/\(.*\) /\1,/')
  $(aws --version | sed 's/ .*//; s#/#,#')
  $(jq --version | sed 's/-/,/')
  $(bb --version | sed 's/ /,/')
  jet,$(jet --version)" > /results/added-pkgs.csv
1.6s
Bash Showcase (Bash)
Bash
0 items
0 items

System Packages and Basics

The packages installed include every major decompression tool, curl, wget, and netcat, gnupg and git, and fontconfig, as well as a small selection of fonts (including the fonts used in the Nextjournal UI).

Packages are installed with apt-get. The environment is saved without cached package lists (they would quickly outdate anyway), and so an update is required before installing any packages. It is recommended to set DEBIAN_FRONTEND=noninteractive for a smooth install.

apt-get -qq update
DEBIAN_FRONTEND=noninteractive \
  apt-get install cowsay fortune
/usr/games/fortune | /usr/games/cowsay
9.5s
Bash Showcase (Bash)
Bash

Displaying Files

Any file written or copied into the /results directory will, once the cell finishes execution, be stored in Nextjournal's immutable, versioned storage database. The editor will attempt to appropriately display files in recognized formats (text, images, CSV tables).

wget -qO /results/test.svg \
  https://nextjournal.com/images/nextjournal-logo.svg
0.2s
Bash Showcase (Bash)
Bash

These files can be accessed in subsequent cells in the same way as Uploaded files, using Ctrl-E or Cmd-E to insert a reference.

ls NJ__REF_
0.7s
Bash Showcase (Bash)
Bash

CSV files will show as paginated tables by default.

echo 'a,b,c
1,2,3' > /results/test.csv
0.1s
Bash Showcase (Bash)
Bash
0 items

Files displayed as text or tables can be changed to show as a downloadable file using the ··· menu to the upper left of the file's display area.

echo 'a b c
1 2 3' > /results/test.txt
0.1s
Bash Showcase (Bash)
Bash
test.txt

Setup

Build a Minimal Bash Environment

The minimal Bash environment is based off the Ubuntu 18.04 LTS Docker image.

Ubuntu Base
Download as Docker image from:
Copy
This image was imported from: ubuntu:18.04

We set the timezone and install some basic packages. We've also set a number of environment variables in the runtime's settings—in particular, the value of NEXTJOURNAL_MOUNT_CUDA sets the default Nvidia CUDA and cuDNN versions for all descendant environments.

The basic packages installed include tools to aid in further package installation, such as decompression suites and network tools.

echo "UTC" > /etc/timezone
echo 'APT::Get::Assume-Yes "true";' > /etc/apt/apt.conf.d/90-yes
apt-get -qq update
DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends \
  apt-utils apt-transport-https \
  ca-certificates locales tzdata sudo \
  zip unzip gzip bzip2 xz-utils unrar p7zip-full \
  curl wget netcat net-tools
apt-get clean
rm -r /var/lib/apt/lists/* # Clear package list so it isn't stale
18.9s
Minimal Bash (Bash)
Ubuntu Base

Generate the locale that we set in the environment variables.

locale-gen en_US.UTF-8
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US.en LC_ALL=en_US.UTF-8
1.3s
Minimal Bash (Bash)
Ubuntu Base

And ensure the GPU libraries can be used.

echo "/usr/local/cuda/lib64" >> /etc/ld.so.conf.d/cuda.conf
echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf
ldconfig
0.4s
Minimal Bash (Bash)
Ubuntu Base

Still tiny.

du -hsx /
0.9s
Minimal Bash (Bash)
Ubuntu Base

Build the default Bash environment

The default Bash environment adds a minimal system Python setup, GCC7 build tools, and various libraries (most notably Apache Arrow) and support tools (including gnupg, git, and cloud-service clients). We also install font tools and some basic fonts.

Install system packages and libraries, development files, and fonts.

apt-get -qq update
4.7s
Bash (Bash)
Minimal Bash
DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends \
  build-essential gfortran cmake automake libtool libltdl-dev pkg-config \
  python3-pip python3-setuptools python3-wheel \
  gnupg git lsb-release patch perl-doc \
  fontconfig fonts-dejavu fonts-liberation2
49.4s
Bash (Bash)
Minimal Bash

Install support libraries for Apache Arrow.

DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends \
  libblas3 liblapack3 libre2-4 libpcre16-3 libpcre32-3 libpcrecpp0v5 \
  libboost-filesystem1.65.1 libboost-system1.65.1 libboost-regex1.65.1 \
  libdouble-conversion1 libelf1 libgflags2.2 libgoogle-glog0v5 libjemalloc1 \
  libgirepository-1.0-1 libffi6 libglib2.0-bin libglib2.0-data \
  zlib1g libbrotli1 libsnappy1v5 libunwind8
7.6s
Bash (Bash)
Minimal Bash

Untar the Flatbuffers library, and the Apache Thrift compiler and Arrow system libraries and tools (includes Parquet, Gandiva, and Plasma support), compiled in this notebook.

FILENAME="apache-arrow+-njbuild-0.15.1.tar.gz"
cd /usr/local
wget --progress=bar:force -O $FILENAME \
  https://nextjournal.com/data/QmbN2YUEM2b3anLTEvEw6wQBGzaESxfCYRGPb4g5Zk1K7y
tar -zxf $FILENAME
rm $FILENAME
3.0s
Bash (Bash)
Minimal Bash

Install Python dependencies for cloud clients.

DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends \
  python3-{botocore,crcmod,colorama,dateutil,docutils,jmespath,pyasn1-modules,rsa,s3transfer,six,urllib3,yaml}
14.8s
Bash (Bash)
Minimal Bash

Update and configure the system pip, and install the AWS command-line client.

printf "[global]\ndisable-pip-version-check = True\n" > /etc/pip.conf
pip3 install --upgrade pip awscli
12.0s
Bash (Bash)
Minimal Bash

Install the Google Cloud SDK.

GCSDK_VERSION="272.0.0"
gcfile="google-cloud-sdk-${GCSDK_VERSION}-linux-x86_64.tar.gz"
cd /usr/local
wget --progress=bar:force \
  https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/${gcfile}
tar -zxf $gcfile
rm $gcfile
cd google-cloud-sdk
bash ./install.sh --quiet --usage-reporting false --command-completion false \
  --path-update false
5.4s
Bash (Bash)
Minimal Bash

Rebuild ld.so.cache, clean up apt, and delete the package list so it doesn't go stale.

ldconfig -v
apt-get clean
rm -r /var/lib/apt/lists/*
1.2s
Bash (Bash)
Minimal Bash

Download and install jq, babashka, and jet. Get latest versions via the Github API.

latest_release () {
  curl -s https://api.github.com/repos/$1/releases | \
    grep tag_name | head -n 1 | sed 's/.*: "\(.\+\)",$/\1/; s/[^[:digit:].]//g'
}
JQ_VERSION=$(latest_release stedolan/jq)
jq_file="jq-linux64"
jq_path="stedolan/jq/releases/download/jq-${JQ_VERSION}/${jq_file}"
jq_url="https://github.com/${jq_path}"
BB_VERSION=$(latest_release borkdude/babashka)
bb_file="babashka-${BB_VERSION}-linux-amd64.zip"
bb_path="borkdude/babashka/releases/download/v${BB_VERSION}/${bb_file}"
bb_url="https://github.com/${bb_path}"
JET_VERSION=$(latest_release borkdude/jet)
jet_file="jet-${JET_VERSION}-linux-amd64.zip"
jet_path="borkdude/jet/releases/download/v${JET_VERSION}/${jet_file}"
jet_url="https://github.com/${jet_path}"
wget --progress=bar:force $jq_url $bb_url $jet_url
mv $jq_file /usr/local/bin/jq
chmod +x /usr/local/bin/jq
unzip $bb_file
mv bb /usr/local/bin/
rm $bb_file
unzip $jet_file
mv jet /usr/local/bin/
rm $jet_file
7.3s
Bash (Bash)
Minimal Bash

Test & print versions.

echo "Name,Version
$(gcloud --version | sed 's/\(.*\) /\1,/')
$(aws --version | sed 's/ .*//; s#/#,#')
$(jq --version | sed 's/-/,/')
$(bb --version | sed 's/ /,/')
jet,$(jet --version)" > /results/ver.csv
1.1s
Bash (Bash)
Minimal Bash
0 items

Install a small selection of open fonts available on Google Fonts and here. The Nextjournal UI uses Fira Sans, Fira Mono, and PT Serif.

nj_fonts.tar.xz
cd /usr/share/fonts/
sudo tar -Jxf NJ__REF_
shopt -s extglob
sudo chmod -R --reference=/usr/share/fonts/truetype/dejavu \
  /usr/share/fonts/!(dejavu)
fc-cache -f
fc-list
4.6s
Bash (Bash)
Minimal Bash

Configure default fonts: Nextjournal UI fonts for the generic families, and similar alternatives for some specific common fonts.

echo '<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
  <alias>
    <family>serif</family>
    <prefer><family>PT Serif</family></prefer>
  </alias>
  <alias>
    <family>sans-serif</family>
    <prefer><family>Fira Sans</family></prefer>
  </alias>
  <alias>
    <family>sans</family>
    <prefer><family>Fira Sans</family></prefer>
  </alias>
  <alias>
    <family>monospace</family>
    <prefer><family>Fira Mono</family></prefer>
  </alias>
  <alias>
    <family>Arial</family>
    <prefer><family>Liberation Sans</family></prefer>
  </alias>
  <alias>
    <family>Helvetica</family>
    <prefer><family>Liberation Sans</family></prefer>
  </alias>
  <alias>
    <family>Verdana</family>
    <prefer><family>Andika</family></prefer>
  </alias>
  <alias>
    <family>Comic Sans</family>
    <prefer><family>Comic Neue</family></prefer>
  </alias>
</fontconfig>' > /etc/fonts/local.conf
0.1s
Bash (Bash)
Minimal Bash

Less tiny.

du -hsx /
1.1s
Bash (Bash)
Minimal Bash

Build a base for Jupyter environments

This will be used as a base for any language environment that needs Python for Jupyter, but not conda. This includes R and all of the Jupyter Runtime Language environments.

Install some commonly needed libraries and development files.

apt-get -qq update
DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends \
  libzmq3-dev libczmq-dev zlib1g-dev libssl-dev libreadline-dev libffi-dev \
  libncurses5-dev libncursesw5-dev libcurl4-openssl-dev \
  libcairo2-dev libpango1.0-dev libpangocairo-1.0
apt-get clean
rm -r /var/lib/apt/lists/* # Clear package list so it isn't stale
28.1s
Jupyter Base (Bash)
Bash

Install Juypter components and dependencies. Using development version of jupyter-core to fix connection file permissions issue—remove when 4.6.2 is released. —MPD 2019.11.24.

pip3 install jupyter jupyter_client \
  git+https://github.com/jupyter/jupyter_core
21.2s
Jupyter Base (Bash)
Bash

We don't actually want a system-Python kernel.

jupyter kernelspec remove -f 'python3'
1.3s
Jupyter Base (Bash)
Bash
du -hsx /
1.3s
Jupyter Base (Bash)
Bash
Runtimes (4)