The Nextjournal Julia Environment
This article builds reusable environments for Julia runtimes, based on minimal Bash environments. The exact versions are
To make use of an environment from this article, follow these steps:
- Create a new article, and insert a Julia cell.
- Activate the runtime settings in the sidebar (see image below).
- Bring up the Environments dropdown.
- Select the Transclude environment… item.
- Search for this article,
nextjournal/julia-environment
, by entering 'julia' into the search field. - Select it to list all environments within.
- Select the desired environment.
1. Showcase
The following packages are included in Nextjournal's Julia 1.1 environment.
1.1. System Packages and Basics
A wide variety of support libraries are installed, as well as gcc v7 and ImageMagick. We also install a static build of FFmpeg, for generating animations.
Version
1.2. Plotting
The default environment comes with GR v
1.2.1. Plots
The JuliaPlots framework provides a unified interface to multiple graphical backends. The default backend is GR, which efficiently generates static plots and animations.
using Plots # define the Lorenz attractor mutable struct Lorenz dt; σ; ρ; β; x; y; z end function step!(l::Lorenz) dx = l.σ*(l.y - l.x) ; l.x += l.dt * dx dy = l.x*(l.ρ - l.z) - l.y ; l.y += l.dt * dy dz = l.x*l.y - l.β*l.z ; l.z += l.dt * dz end attractor = Lorenz((dt = 0.02, σ = 10., ρ = 28., β = 8//3, x = 1., y = 1., z = 1.)...) # initialize a 3D plot with 1 empty series plt = plot3d(1, xlim=(-25,25), ylim=(-25,25), zlim=(0,50), title = "Lorenz Attractor", marker = 2) # build an animated gif by pushing new points to the plot, saving every 10th frame anim = for i=1:1500 step!(attractor) push!(plt, attractor.x, attractor.y, attractor.z) end every 10 gif(anim,"/results/anim.gif")
Switching to the Plotly backend adds some interactivity to the output.
plotly(lw=3) x = -100:100 Plots.plot(x, 100x.^2) Plots.plot!(x, x.^3 - x.^2)
1.2.2. Makie
The Makie package can use a GPU to quickly generate beautiful visualizations.
using Makie, LinearAlgebra n = 20 f = (x,y,z) -> x*exp(cos(y)*z) ∇f = (x,y,z) -> Point3f0(exp(cos(y)*z), -sin(y)*z*x*exp(cos(y)*z), x*cos(y)*exp(cos(y)*z)) ∇ˢf = (x,y,z) -> ∇f(x,y,z) - Point3f0(x,y,z)*dot(Point3f0(x,y,z), ∇f(x,y,z)) θ = [0;(0.5:n-0.5)/n;1] φ = [(0:2n-2)*2/(2n-1);2] x = [cospi(φ)*sinpi(θ) for θ in θ, φ in φ] y = [sinpi(φ)*sinpi(θ) for θ in θ, φ in φ] z = [cospi(θ) for θ in θ, φ in φ] pts = vec(Point3f0.(x, y, z)) ∇ˢF = vec(∇ˢf.(x, y, z)) .* 0.1f0 Makie.surface(x, y, z, resolution = (800,800)) arrows!(pts, ∇ˢF, arrowsize = 0.03, linecolor = (:white, 0.6), linewidth = 3)
1.3. Data Structures
Several data-related packages are installed in the default environment.
- The FileIO v
- DataFrames v
1.3.1. File Handling
The HDF5 package provides a Julia interface to the HDF5 library. It is also used by the JLD and MAT packages. Let's look at some example temperature data.
using HDF5 # Some methods to traverse the first path in an h5 file. dd(node::HDF5File) = dd(node[names(node)[1]]) dd(node::HDF5Group) = dd(node[names(node)[1]]) dd(node::HDF5Dataset) = node dspath = h5open(NEONDSTowerTemperatureData.hdf5) do h5 name(dd(h5)) end print("First path: $dspath.") data = h5read(NEONDSTowerTemperatureData.hdf5, dspath) using Plots Plots.plot([x.data[1] for x in data],[y.data[3] for y in data], xrotation=45,legend=:none)
The JLD package provides a type-preserving way to save Julia objects to file.
using JLD struct testData x::Int64 y::String end foo = testData(7,"test") save("/tmp/blah.jld", "foo", foo) load("/tmp/blah.jld")
1.3.2. FileIO
FileIO provides a unified set of methods to access data in files: query()
, load()
, and save()
, as well as loadstreaming()
and savestreaming()
for large files. The functions can automatically recognize files using headers or extensions, but you can also provide format information as below, where we load the Iris Dataset from a CSV file.
using FileIO load(File(format"CSV",iris.csv))
1.3.3. DataFrames
DataFrame objects represent tabular data as a set of vectors.
using DataFrames df = DataFrame(A = 1:5, B = 1:2:10, C = ["a","b","c","d","q"])
"A" | "B" | "C" |
---|---|---|
1 | 1 | "a" |
2 | 3 | "b" |
3 | 5 | "c" |
4 | 7 | "d" |
5 | 9 | "q" |
Column names are referenced with Julia Symbols, not strings.
df[:A] + df[Symbol("B")]
1.3.4. JSON
Import and export JSON using the JSON package, which is always loaded on Nextjournal. In the example below, a Julia data structure input results in JSON output. The change from nothing
to null
is a clear indicator.
json(["foo", Dict("bar" => ("baz", nothing, 1.0, 2))])
2. Setup
2.1. Julia 1.1
2.1.1. Build a Minimal Julia 1.1 Environment
We'll base our environment off of our Minimal Bash image. Note that the Julia version is set as an environment variable on the runtime.
PATH | /usr/local/julia/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin |
JULIA_PATH | /usr/local/julia |
JULIA_VERSION | 1.1.1 |
The exact version we're installing is
tarArch='x86_64' dirArch='x64' JULIA_VERSION_SHORT=${JULIA_VERSION/-rc/} FILENAME="julia-${JULIA_VERSION}-linux-${tarArch}.tar.gz" FILEURL="https://julialang-s3.julialang.org/bin/linux/${dirArch}/${JULIA_VERSION_SHORT%[.-]*}/${FILENAME}" echo "Downloading ${FILEURL}." curl -fL -o julia.sha256 \ "https://julialang-s3.julialang.org/bin/checksums/julia-${JULIA_VERSION}.sha256" curl -fL -o julia.tar.gz.asc "${FILEURL}.asc" curl -fL -o julia.tar.gz "${FILEURL}" echo `grep $FILENAME julia.sha256 | cut -d " " -f1` > j256sig
Check the signatures to verify our download.
sha256=`cat j256sig` echo "${sha256} *julia.tar.gz" | sha256sum -c - export GNUPGHOME="$(mktemp -d)" JULIA_GPG="3673DF529D9049477F76B37566E3C7DC03D6E495" gpg --keyserver ha.pool.sks-keyservers.net --recv-keys "$JULIA_GPG" gpg --batch --verify julia.tar.gz.asc julia.tar.gz command -v gpgconf > /dev/null && gpgconf --kill all rm -rf "$GNUPGHOME" julia.tar.gz.asc julia.sha256
Install and remove the archive.
mkdir -p "$JULIA_PATH" tar -xzf julia.tar.gz -C "$JULIA_PATH" --strip-components 1 rm julia.tar.gz j256sig
And verify it runs.
julia -v
The JSON package is required to run on Nextjournal.
julia -E 'using Pkg pkg"update" pkg"add JSON" pkg"build" pkg"precompile"'
2.1.2. Build the Default Julia 1.1 Environment
We'll add a number of packages as well as the libraries and support programs required by them. The major packages installed are JuliaPlots along with the GR and PlotlyJS backends, Makie, and the DataFrames, CSV, and HDF5 data-handling packages. The setup requires a GPU node for Makie to successfully precompile, but not to transclude and use unless Makie itself is needed.
First we'll check that the minimal Julia env works.
"$VERSION"
Some Julia packages require gcc to compile, so first we'll install that, as well as various required tools and libraries.
apt-get -qq update apt-get install --no-install-recommends \ imagemagick libhdf5-dev hdf5-tools mesa-utils \ build-essential gfortran cmake automake libtool libltdl-dev pkg-config \ libxt6 libxrender1 libgl1-mesa-glx libqt5widgets5 `# for GR` \ libhttp-parser2.7.1 `# for PlotlyJS` \ libxrandr-dev libxinerama-dev libxcursor-dev libglfw3-dev `# for Makie` apt-get clean rm -r /var/lib/apt/lists/* # Clear package list so it isn't stale
Plots also needs FFmpeg to create animations, so we'll install a 64-bit static build from an install package we download in the Appendix.
IDIR="/usr/local/ffmpeg" mkdir -p $IDIR cd $IDIR tar -Jxfffmpeg-release-amd64-static.tar.xz--strip 1 for ex in $IDIR/{ffmpeg,ffmpeg-10bit,ffprobe,qt-faststart}; do ln -s $ex /usr/local/bin/ done
Next the Julia package installs. Bugfix: Makie release fails to precompile, recommendation is using #master for Makie, GLMakie, and AbstractPlotting—mpd, 9 Dec 2018.
using Pkg pkg"update" pkg"add SoftGlobalScope DataFrames JLD CSV CSVFiles Netpbm NRRD MeshIO HDF5 MAT FileIO JSExpr CSSUtil StatsBase StatsPlots Observables Interact WebSockets HTTP Blink WebIO PlotlyBase PlotlyJS RecipesBase GR Plots ImageCore ImageShow ImageMagick Colors BenchmarkTools FixedPointNumbers AbstractPlotting#master GLMakie#master Makie#master IJulia"
Build all packages.
pkg"build"
And finally precompilation of any qualifying packages.
pkg"precompile"
Finally, add font defaults for JuliaPlots and Makie. These named Code Listings will be mounted as files to the runtime's filesystem, and saved with the environment.
PLOTS_DEFAULTS = Dict(:fontfamily => "Open Sans")
Attributes(font = "Open Sans")
2.1.3. Test Default 1.1 Env
using SoftGlobalScope, DataFrames, JLD, CSV, CSVFiles, Netpbm, NRRD, MeshIO, HDF5, MAT, FileIO, JSExpr, CSSUtil, StatsBase, StatsPlots, Observables, Interact, WebSockets, HTTP, Blink, WebIO, PlotlyBase, PlotlyJS, RecipesBase, GR, Plots, ImageCore, ImageShow, ImageMagick, Colors, Makie, BenchmarkTools, FixedPointNumbers, IJulia IJulia.verbose
"$(Pkg.installed()["SoftGlobalScope"])"
"$(Pkg.installed()["GR"])"
"$(Pkg.installed()["PlotlyBase"])"
"$(Pkg.installed()["PlotlyJS"])"
"$(Pkg.installed()["Plots"])"
"$(Pkg.installed()["Makie"])"
"$(Pkg.installed()["JLD"])"
"$(Pkg.installed()["CSV"])"
"$(Pkg.installed()["HDF5"])"
"$(Pkg.installed()["JSON"])"
"$(Pkg.installed()["FileIO"])"
"$(Pkg.installed()["DataFrames"])"
2.2. Julia 0.7
2.2.1. Julia 0.7 Minimal
We're installing version
2.2.2. Julia 0.7 Default
"$VERSION"
Fixing Plots at v0.19.3, the current final 0.7 version. This seems to clear up some other version/dep issues. —mpd, 16 Dec 2018.
using Pkg pkg"update" pkg"add SoftGlobalScope DataFrames JLD CSV CSVFiles Netpbm NRRD MeshIO HDF5 MAT FileIO JSExpr CSSUtil StatsBase StatsPlots Observables Interact WebSockets HTTP Blink WebIO PlotlyBase PlotlyJS RecipesBase GR Plots#v0.19.3 ImageCore ImageShow ImageMagick Colors BenchmarkTools FixedPointNumbers AbstractPlotting GLMakie Makie IJulia"
pkg"build"
pkg"precompile"
2.2.3. Test Default 0.7 Env
using SoftGlobalScope, DataFrames, JLD, CSV, CSVFiles, Netpbm, NRRD, MeshIO, HDF5, MAT, FileIO, JSExpr, CSSUtil, StatsBase, StatsPlots, Observables, Interact, WebSockets, HTTP, Blink, WebIO, PlotlyBase, PlotlyJS, RecipesBase, GR, Plots, ImageCore, ImageShow, ImageMagick, Colors, BenchmarkTools, FixedPointNumbers, Makie, IJulia IJulia.verbose