Transclusions

Luis Deliz, Water, 2011, photograph

A transclusion is a special type of immutable reference to content in another article. Their immutability means that state cannot be changed, eliminating common problems associated with hyperlinks and modifying shared assets.

Nextjournal's environment transclusions are a sane way of sharing a single technology stack between multiple articles. This offers several specific benefits:

  • An update to an environment creates a new environment. Nextjournal stores the old one, automatically versioning your changes along the way.
  • Non-destructive changes make time travel simple, eliminating risk when experimenting with new dependencies.
  • Transcluded environments always convey their origins. Origin articles provide context for decisions made at arbitrary points in the past.
  • Changes to the original environment will not affect transcluded environments. Authors can make changes knowing they will not break articles that have transcluded their work.

Since Nextjournal automatically saves the exact version of all dependencies, transcluded environments offer the promise of equivalency - an essential step towards rigorous reproducibility and a way to engender collaboration through ease and confidence.

1.
Understanding Transclusions

The difference between a transclusion and a conventional reference like hyperlinks is nuanced. Rather than refer to content in another article, transclusions include content from another article.

Ted Nelson, the person that coined the term, describes the unique fluidity of interconnections offered by transclusions in this clip from Warner Herzog's Lo and Behold (2016):

2.
How to Transclude Environments

An environment is a filesystem snapshot of the runtime. It includes everything that is necessary to run whatever is in its associated code cells - from the operating system to the smallest dependency.

In the example below, an environment called Color Palette is built in an article called Color Palette Extractor. The author installed two packages, pillow for image manipulation and haishoku for the image color analysis, and added some contextual information for human readers.

2.1.
Export the Environment

2.1.1.
Step 1: Export the Environment

  • Select the gear icon to the right of the runtime to open the runtime settings.
  • Check the box labeled Export the environment.

Export the environment

2.1.2.
Step 2: Publish the Article

Create a URL that's representative of your article. This will be how you find the transcluded environment in the future. For example, Color Palette Extractor is published at color-extractor.

2.2.
Transcluding an Environment

The environment created in Color Palette Extractor will now be available to all your other articles. This includes other articles shared by any groups you belong to.

2.2.1.
Step 1: Create a New Runtime

Create a new runtime

2.2.2.
Step 2: Select a New Environment

Select the gear icon to the right of the runtime to open the runtime settings and then select the Environment dropdown menu.

Open the settings for your new runtime

Scroll down to Transclude environment.

Transclude environment

Type in or scroll to the name of your article, in this case color-extractor.

Select the article with the environment you want to transclude

Now select Color Palette from the list of available environments.

Select the environment to transclude

As you can see in The Hues of Turner article, all the packages available in Color Palette Extractor are immediately available in the new article - no installation required!

3.
How Transclusions Work

The Hues of Turner is a modest study in the wide-ranging field of art, culture, and the use of color. To help reveal larger correlations, the author, Sandra, collaborates with a machine learning expert, Abduba.

Abduba uses Nextjournal's Tensorflow template and transcludes Sandra's environment. He can immediately get to work analyzing the color space for tens of thousands of images in a new article.

Remember that a transclusion is not a simple reference, it is an entirely new copy that points back to its origin. This subtle detail means Sandra can refine the tooling without worrying about breaking Abduba's work and vice versa.

Abduba transcluding Sandra's environment

The top track depicts Sandra's timeline with Abduba just below. Sandra publishes her environment as color-extractor and Abduba transcludes it as a basis for his article, abduba/ML.

Sandra updates color-extractor twice: once to remove unused dependencies and once to add new dependencies. Each change creates a new Docker image with a unique SHA key. From Sandra's perspective they are all color-extractor, from Nextjournal's perspective there exists three Docker images in time. If Sandra makes an undesired change at any point, it's simple to revert to any point in the past.

Abduba can complete his task using the original dependencies, so he publishes using the original color-extractor environment (addressed as SHA 111). The diagram above illustrates two important features of Nextjournal's environments:

  • Changes to Sandra's environment do not affect Abduba's work. This is a core benefit of transclusion technology.
  • Changes in Abduba's article and associated data do not alter its environment. Articles, data, and environments are fundamentally decoupled.

The latter allows Nextjournal to automatically version control your environments, articles, and data separately. This means Nextjournal creates a fully history on your behalf while you focus on your work!

4.
More Than a Notebook

Nextjournal promises complete reproducibility across your entire project. Read the introduction to environments to learn more about how this enables experimentation and sharing. Refer to the introduction to remixing to learn more about how the platform enables collaboration. For a broad overview what what makes Nextjournal different from other notebooks and development environments, check out Why Nextjournal. If you'd like to try it yourself, get in touch for Private Beta access.

© 2018 Nextjournal GmbH