The Nextjournal Python Environment
Nexjournal's Python 3 environment runs version
Learn more about environments on Nextjournal.
1. Showcase
These packages are included in Nextjournal's Python 3 environment.
1.1. Package Management
Packages are installed using conda
or pip
version setuptools
version
1.2. Plotting
The default environment comes with plotly
version matplotlib
version
1.2.1. Plotly
Plot a histogram using plotly
, a plotting library for making interactive graphs online.
import plotly.graph_objs as go import numpy as np x0 = np.random.randn(500) x1 = np.random.randn(500)+1 trace1 = go.Histogram(x=x0, opacity=0.75) trace2 = go.Histogram(x=x1, opacity=0.75) layout = go.Layout(barmode='overlay') go.Figure(data=[trace1, trace2], layout=layout)
1.2.2. Matplotlib
Plot a matplotlib
, a Python plotting library.
import matplotlib.pyplot as plt # Data for plotting t = np.arange(0.0, 2.0, 0.01) s = 1 + np.sin((sine-cycles* 2)* np.pi * t) # Note that using plt.subplots below is equivalent to using # fig = plt.figure() and then ax = fig.add_subplot(111) fig, ax = plt.subplots() ax.plot(t, s) ax.set(xlabel='time (s)', ylabel='voltage (mV)', title='Sine Wave') ax.grid() fig
1.3. Data Structures
Nextjournal's default Python environment contains several packages for data manipulation and parsing.
- The SciPy ecosystem is available, including
scipy
versionnumpy
versionpandas
version simplejson
versionsix
version
1.3.1. Numpy
Numpy
's main object is a N-dimensional array useful for linear algebra, Fourier transforms, and random number capabilities. Here it is used to create a Mandelbrot set which is ultimately plotted using matplotlib
.
def mandelbrot( h,w, maxit=10): y,x = np.ogrid[ -1.4:1.4:h*1j, -2:0.8:w*1j ] c = x+y*1j z = c divtime = maxit + np.zeros(z.shape, dtype=int) for i in range(maxit): z = z**2 + c diverge = z * np.conj(z) > 2**2 # who is diverging div_now = diverge & (divtime==maxit) # who is diverging now divtime[div_now] = i + 100 # note when z[diverge] = 2 # avoid diverging too much return divtime fig = plt.subplots(1,figsize=(20,20)) plt.imshow(mandelbrot(1000,1000)) plt.axis('off') plt.savefig("/results/temp.png")
1.3.2. Pandas
Pandas
makes data analysis easier with Python. For example, a single instantiation of pandas
' Series
class can include all label and data information. numpy
and the final graph is plotted with matplotlib
.
import pandas as pd ts = pd.Series(np.random.randn(pandas), index=pd.date_range('1/1/2000', periods=pandas)) ts = ts.cumsum() fig, ax = plt.subplots() ax = ts.plot() fig
1.3.3. Simplejson
Import and export JSON on Nextjournal using simplejson
. In the example below, a Python data structure input results in JSON output. The change from None
to null
is a clear indicator.
import simplejson as json json.dumps(['foo', {'bar': ('baz', None, 1.0, 2)}])
1.3.4. Six
Six
makes it easy to write Python code that is compatible with both Python 2 and Python 3.
For example, Python 2's urllib
, urllib2
, and urlparse
modules have been combined in the urllib
package in Python 3. The six.moves.urllib
package is a version-independent location for this functionality.
Python 2:
from six.moves.urllib.request import urlopen url = urlopen("http://nextjournal.com") print url.read()
Python 3:
from six.moves.urllib.request import urlopen url = urlopen("http://nextjournal.com") print(url.read())
2. Setup
2.1. Build a Minimal Python 3 Environment
Download and install conda
.
CONDA_VER="4.5.4" curl -sSL -o ~/anaconda.sh \ https://repo.continuum.io/miniconda/Miniconda3-${CONDA_VER}-Linux-x86_64.sh /bin/bash ~/anaconda.sh -b -p /opt/conda rm ~/anaconda.sh
Add conda
's library directory so ldconfig
will pick it up, set conda config
, and ensure pip
is reasonably updated. We also pin Python to the installed minor version, allowing only patch-version up/downgrades.
echo "/opt/conda/lib" >> /etc/ld.so.conf.d/conda.conf conda config --set always_yes True printf "[global]\ndisable-pip-version-check = True\n" > /etc/pip.conf echo 'pip >=18.1' > /opt/conda/conda-meta/pinned # prevent pip downgrade # upgrade Python within minor version PYTHON_MINOR=`python --version 2>&1 | sed 's/Python //;s/.[0-9] ::.*//;'` echo "python =$PYTHON_MINOR" >> /opt/conda/conda-meta/pinned conda update python conda clean -qtipy ldconfig python -V pip -V
2.2. Build the Default Python 3 Environment
2.2.1. Install
We'll install gcc
and other build tools, since some pip
installs require compilation.
apt-get -qq update apt-get install \ build-essential gfortran cmake automake libtool libltdl-dev pkg-config \ libxext6 libhdf5-100 apt-get clean rm -r /var/lib/apt/lists/* # Clear package list so it isn't stale
This default image has support for a number of general-use packages, including pandas
, scipy
, scikit-learn
, scikit-image
, and opencv-python
. For graphical output, matplotlib
and plotly
are installed. We'll also install some basic utilities, as well as setuptools
to make any additional installs less difficult. We're installing Jedi to have code completions for Python, and Jupyter to support notebook imports .
conda install \ setuptools six simplejson dill pillow pytables h5py \ plotly matplotlib tqdm termcolor tabulate \ python-dateutil more-itertools toolz cython cffi attrs decorator jedi \ numpy scipy pandas scikit-learn scikit-image \ jupyter conda clean -qtipy ldconfig python -V pip -V jupyter --version jupyter kernelspec list jupyter --paths
And we'll install the unofficial wheel of OpenCV.
pip install opencv-python-headless
Finally, set up default fonts for matplotlib
.
mkdir -p ~/.config/matplotlib/ echo 'font.family: sans-serif font.sans-serif: Fira Sans, PT Sans, Open Sans, Roboto, DejaVu Sans, Liberation Sans, sans-serif font.serif: PT Serif, Noto Serif, DejaVu Serif, Liberation Serif, serif font.monospace: Fira Mono, Roboto Mono, DejaVu Sans Mono, Liberation Mono, Fixed, Terminal, monospace' > ~/.config/matplotlib/matplotlibrc
2.2.2. Test
python --version
jupyter kernelspec list jupyter --version jupyter --paths
import platform; platform.python_version()
import pip; pip.__version__
import plotly; plotly.__version__
import numpy as np; np.__version__
import matplotlib; matplotlib.__version__
import setuptools; setuptools.__version__
import six; six.__version__
import simplejson; simplejson.__version__
import pandas; pandas.__version__
import scipy; scipy.__version__
2.3. Minimal Python 2
Download and install conda
.
CONDA_VER="4.5.4" curl -sSL -o ~/anaconda.sh \ https://repo.continuum.io/miniconda/Miniconda2-${CONDA_VER}-Linux-x86_64.sh /bin/bash ~/anaconda.sh -b -p /opt/conda rm ~/anaconda.sh
Setup conda
, ld
, and pip
.
2.4. Default Python 2
2.4.1. Install
2.4.2. Test
python --version
jupyter kernelspec list jupyter --version jupyter --paths
import platform; platform.python_version()