The Nextjournal Julia Environment

This article builds reusable environments for Julia runtimes, based on the minimal Bash environment. Nextjournal's Julia 1.1 environment runs version , while the 0.7 environment runs version , both installed from the generic Linux binary packages.

Learn more about environments on Nextjournal.

You can quickly make use of the Julia 1.1 environment by remixing the Nextjournal Julia template, or use these environments with any existing runtime by following these steps (see image below):

Activate the runtime settings in the sidebar.
Bring up the Environments dropdown.
Select Import environment… at the bottom of the list.
Search for this article, nextjournal/julia-environment.
Select it to list all environments within.
Select the desired environment.

1. Showcase

These packages are included in Nextjournal's Julia 1.1 environment.

pkg"status"

0.7s

Julia 1.3-alpha Test (Julia)

1.1. System Packages and Basics

A wide variety of support libraries are installed, as well as gcc v7 and ImageMagick.

Version of the SoftGlobalScope package is available—this adjusts scope-handling to be more like Julia 0.6, providing a more REPL-like interface.

Julia packages are installed normally, using Pkg in a Julia cell. Please refer to the Julia section of Installing Software and Packages for more detailed information.

1.2. Plotting

The default environment comes with GR v , PlotlyBase v , and PlotlyJS v , as well as version of the Plots framework. Makie version is also installed.

1.2.1. Plots

The JuliaPlots framework provides a unified interface to multiple graphical backends. The default backend is GR, which efficiently generates static plots and animations.

using Plots

# define the Lorenz attractor
mutable struct Lorenz
    dt; σ; ρ; β; x; y; z
end

function step!(l::Lorenz)
    dx = l.σ*(l.y - l.x)       ; l.x += l.dt * dx
    dy = l.x*(l.ρ - l.z) - l.y ; l.y += l.dt * dy
    dz = l.x*l.y - l.β*l.z     ; l.z += l.dt * dz
end

attractor = 
  Lorenz((dt = 0.02, σ = 10., ρ = 28., β = 8//3, x = 1., y = 1., z = 1.)...)

# initialize a 3D plot with 1 empty series
plt = plot3d(1, xlim=(-25,25), ylim=(-25,25), zlim=(0,50),
             title = "Lorenz Attractor", marker = 2)

# build an animated gif by pushing new points to the plot, saving every 10th frame
anim = @animate for i=1:1500
    step!(attractor)
    push!(plt, attractor.x, attractor.y, attractor.z)
end every 10

gif(anim,"/results/anim.gif")

50.3s

Julia 1.3-alpha Test (Julia)

Switching to the Plotly backend adds some interactivity to the output.

plotly(lw=3)
x = -100:100

Plots.plot(x, 100x.^2)
Plots.plot!(x, x.^3 - x.^2)

7.9s

Julia 1.3-alpha Test (Julia)

1.2.2. Makie

The Makie package can use a GPU to quickly generate beautiful visualizations.

using Makie, LinearAlgebra

n = 20
f   = (x,y,z) -> x*exp(cos(y)*z)
∇f  = (x,y,z) -> Point3f0(exp(cos(y)*z), -sin(y)*z*x*exp(cos(y)*z), x*cos(y)*exp(cos(y)*z))
∇ˢf = (x,y,z) -> ∇f(x,y,z) - Point3f0(x,y,z)*dot(Point3f0(x,y,z), ∇f(x,y,z))

θ = [0;(0.5:n-0.5)/n;1]
φ = [(0:2n-2)*2/(2n-1);2]
x = [cospi(φ)*sinpi(θ) for θ in θ, φ in φ]
y = [sinpi(φ)*sinpi(θ) for θ in θ, φ in φ]
z = [cospi(θ) for θ in θ, φ in φ]

pts = vec(Point3f0.(x, y, z))
∇ˢF = vec(∇ˢf.(x, y, z)) .* 0.1f0
Makie.surface(x, y, z, resolution = (800,800))
arrows!(pts, ∇ˢF, arrowsize = 0.03, linecolor = (:white, 0.6), linewidth = 3)

68.4s

Julia 1.3-alpha Test (Julia)

Scene (960px, 540px): events: window_area: HyperRectangle{2,Int64}([0, 0], [0, 0]) window_dpi: 100.0 window_open: false mousebuttons: Set(AbstractPlotting.Mouse.Button[]) mouseposition: (0.0, 0.0) mousedrag: notpressed scroll: (0.0, 0.0) keyboardbuttons: Set(AbstractPlotting.Keyboard.Button[]) unicode_input: Char[] dropped_files: String[] hasfocus: false entered_window: false plots: *Axis3D{...} *Surface{...} *Arrows{...} subscenes:

1.3. Data Structures

Several data-related packages are installed in the default environment.

For handling their relative datatypes and I/O for associated files, we install a variety of packages, including JLD v , CSV v , HDF5 v , and JSON v .
The FileIO v framework provides a unified interface for loading files of many types via multiple backends, including ImageMagick.
DataFrames v defines and handles objects similar to those found in R and the Python pandas toolkit, with a comparable interface.

1.3.1. File Handling

The HDF5 package provides a Julia interface to the HDF5 library. It is also used by the JLD and MAT packages. Let's look at some example temperature data.

NEONDSTowerTemperatureData.hdf5

using HDF5

# Some methods to traverse the first path in an h5 file.
dd(node::HDF5File) = dd(node[names(node)[1]])
dd(node::HDF5Group) = dd(node[names(node)[1]])
dd(node::HDF5Dataset) = node

dspath = h5open(NEONDSTowerTemperatureData.hdf5
) do h5
  name(dd(h5))
end
print("First path: $dspath.")

data = h5read(NEONDSTowerTemperatureData.hdf5
, dspath)

using Plots

Plots.plot([x.data[1] for x in data],[y.data[3] for y in data],
  xrotation=45,legend=:none)

11.1s

Julia 1.3-alpha Test (Julia)

The JLD package provides a type-preserving way to save Julia objects to file.

using JLD

struct testData
  x::Int64
  y::String
end

foo = testData(7,"test")

save("/tmp/blah.jld", "foo", foo)
load("/tmp/blah.jld")

14.7s

Julia 1.3-alpha Test (Julia)

Dict{String,Any} with 1 entry: "foo" => testData(7, "test")

1.3.2. FileIO

FileIO provides a unified set of methods to access data in files: query(), load(), and save(), as well as loadstreaming() and savestreaming() for large files. The functions can automatically recognize files using headers or extensions, but you can also provide format information as below, where we load the Iris Dataset from a CSV file.

iris.csv

using FileIO

load(File(format"CSV",iris.csv
))

18.3s

Julia 1.3-alpha Test (Julia)

1.3.3. DataFrames

DataFrame objects represent tabular data as a set of vectors.

using DataFrames

df = DataFrame(A = 1:5, B = 1:2:10, C = ["a","b","c","d","q"])

10.0s

Julia 1.3-alpha Test (Julia)

A	B	C
1	1	a
2	3	b
3	5	c
4	7	d
5	9	q

5 items

Column names are referenced with Julia Symbols, not strings.

df[:A] + df[Symbol("B")]

3.2s

Julia 1.3-alpha Test (Julia)

5-element Array{Int64,1}: 2 5 8 11 14

1.3.4. JSON

Import and export JSON using the JSON package, which is always loaded on Nextjournal. In the example below, a Julia data structure input results in JSON output. The change from nothing to null is a clear indicator.

json(["foo", Dict("bar" => ("baz", nothing, 1.0, 2))])

0.8s

Julia 1.3-alpha Test (Julia)

"[\"foo\",{\"bar\":[\"baz\",null,1.0,2]}]"

2. Setup

2.1. Julia 1.1

2.1.1. Build a Minimal Julia 1.1 Environment

We'll base our environment off of our Minimal Bash image. Note that the Julia version is set as an environment variable on the runtime.

Minimal Julia 1.3-alpha↩

▼

Minimal Julia 1.3-alpha (Bash)

exporting environment

Type: Nextjournal

Environment:

Resources:

Environment Variables:

PATH	/usr/local/julia/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
JULIA_PATH	/usr/local/julia
JULIA_VERSION	1.3.0-alpha

Download this environment as a Docker image from:

The exact version we're installing is . Here we download the archive and signatures.

tarArch='x86_64'
dirArch='x64'
JULIA_VERSION_SHORT=${JULIA_VERSION/-rc/}
FILENAME="julia-${JULIA_VERSION}-linux-${tarArch}.tar.gz"
FILEURL="https://julialangnightlies-s3.julialang.org/bin/linux/x64/julia-latest-linux64.tar.gz"

echo "Downloading ${FILEURL}."

curl -fL -o julia.sha256 \
  "https://julialang-s3.julialang.org/bin/checksums/julia-${JULIA_VERSION}.sha256"
curl -fL -o julia.tar.gz.asc "${FILEURL}.asc"
curl -fL -o julia.tar.gz "${FILEURL}"

echo `grep $FILENAME julia.sha256 | cut -d " " -f1` > j256sig

5.1s

Minimal Julia 1.3-alpha (Bash)

Check the signatures to verify our download.

sha256=`cat j256sig`
echo "${sha256} *julia.tar.gz" | sha256sum -c -

export GNUPGHOME="$(mktemp -d)"
JULIA_GPG="3673DF529D9049477F76B37566E3C7DC03D6E495"
gpg --keyserver ha.pool.sks-keyservers.net --recv-keys "$JULIA_GPG"
gpg --batch --verify julia.tar.gz.asc julia.tar.gz
command -v gpgconf > /dev/null && gpgconf --kill all
rm -rf "$GNUPGHOME" julia.tar.gz.asc julia.sha256

2.2s

Minimal Julia 1.3-alpha (Bash)

Install and remove the archive.

mkdir -p "$JULIA_PATH"
tar -xzf julia.tar.gz -C "$JULIA_PATH" --strip-components 1
rm julia.tar.gz j256sig

5.0s

Minimal Julia 1.3-alpha (Bash)

And verify it runs.

julia -e "using InteractiveUtils; versioninfo(stdout, verbose = true)"

2.7s

Minimal Julia 1.3-alpha (Bash)

The JSON package is required to run on Nextjournal.

julia -E 'using Pkg

pkg"update"
pkg"add JSON"
pkg"build"
pkg"precompile"'

21.4s

Minimal Julia 1.3-alpha (Bash)

2.1.2. Build the Default Julia 1.1 Environment

We'll add a number of packages as well as the libraries and support programs required by them. The major packages installed are JuliaPlots along with the GR and PlotlyJS backends, Makie, and the DataFrames, CSV, and HDF5 data-handling packages. The setup requires a GPU node for Makie to successfully precompile, but not to transclude and use unless Makie itself is needed.

First we'll check that the minimal Julia env works.

"$VERSION"

0.6s

Julia 1.x versionJulia 1.3-alpha (Julia)

"1.3.0-alpha.51"

Some Julia packages require gcc to compile, so first we'll install that, as well as various required tools and libraries.

apt-get -qq update
apt-get install --no-install-recommends \
  libhdf5-dev hdf5-tools \
  build-essential gfortran cmake automake libtool libltdl-dev pkg-config \
  libxt6 libxrender1 libgl1-mesa-glx libqt5widgets5 `# for GR` \
  libhttp-parser2.7.1 `# for PlotlyJS`
apt-get clean
rm -r /var/lib/apt/lists/* # Clear package list so it isn't stale

46.5s

Julia 1.3-alpha (Bash in Julia)

Next the Julia package installs. Bugfix: Makie release fails to precompile, recommendation is using #master for Makie, GLMakie, and AbstractPlotting—mpd, 9 Dec 2018.

using Pkg

pkg"update"
pkg"add SoftGlobalScope DataFrames JLD CSV CSVFiles Netpbm NRRD MeshIO HDF5 MAT FileIO JSExpr CSSUtil StatsBase StatsPlots Observables Interact WebSockets HTTP Blink WebIO PlotlyBase PlotlyJS RecipesBase GR Plots ImageCore ImageShow ImageMagick Colors BenchmarkTools FixedPointNumbers AbstractPlotting IJulia"

177.7s

Julia 1.3-alpha (Julia)

And finally precompilation of any qualifying packages.

pkg"precompile"

403.4s

Julia 1.3-alpha (Julia)

Finally, add font defaults for JuliaPlots and Makie. These named Code Listings will be mounted as files to the runtime's filesystem, and saved with the environment.

PLOTS_DEFAULTS = Dict(:fontfamily => "Open Sans")

.juliarc.jl

Attributes(font = "Open Sans")

theme.jl

Testing Julia environment:

Pkg.status()

0.6s

Julia 1.3-alpha Test (Julia)