Say What You Mean

Exploring Declarative Computation in Art

Me

  • 🎓 Computer Science → Music

  • 🎥 Time-Based Media

  • 💻 Nextjournal

@dschmudde . http://schmud.de/

Artists + Programmers

Artists and programmers are in the business of representation and abstraction.

Contemporary Art

Maurizio Cattelan Untitled (all five horses together at once, for the first time) (2013)

Object

The human scale.inline_formula not implemented

What does it mean or what does it do?

Environment

The scale of reality.

Can it be manipulated?

Art

← 19th Century Art

Still-Life with an Aquamanile, Fruit, and a Nautilus Cup Willem Kalf (c. 1660)

  • Rendering accuracy

  • Dramatic depictions

Still Life with Flowers, Fan, and Pearls Edouard Manet (c. 1860)

  • Depiction of the commonplace

  • Quality of the object's attributes (color, texture, ...)

→ 20th Century Art

Still Life With Gingerpot I Piet Mondrian (1911/12)

  • The medium of painting itself

  • The artist's metaphysical state

White Wraparound Triptych (blue, green, lavender) Jo Baer (1970-1974)

Contemporary Art

The art object intrinsically feels more knowable and controlled than the conditions that determine how art is experienced. The environment.

Models

The Computer as a Communication Device

J.C.R. Licklider and Robert W. Taylor (1968)

Computer Modeling → Communication

  • Insight into the behavior of a system → insight into another person's mind.

  • Includes presumptions, unconscious biases, factual failings, and intuitive intelligence.

Hubris in Modeling

As mid-century artists acknowledge the greater forces of the environment, computer scientists endeavor to model the unknown and promise big returns.

  • Natural language processing of the 1960s

  • The Computer as a Communication Device

Object Oriented Programming

The model is the code. The object is the state.

class Post:
    __max_size = 280
    text = ""
    def __init__(self, text):
        self.text = text
    def __validate_size(self):
        if(len(self.text) <= self.__max_size):
            return True
        else:
            return False
    def send(self):
        if self.__validate_size():
            return "API CALL: " + self.text
        else:
            return "API CALL: Error - text too long"
0.2s
tweet = Post("lorem ipsum")
tweet.send()
0.3s
tweet = Post(" lorem ipsum lorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsumlorem ipsum")
tweet.send()
0.2s

Specifications

The model validates the code. The data is the state.

Function

(ns user)
(defn api-call [text video]
  (str "API CALL: " text " + " video))
(defn send-post [text & opts] 
   (let [{:keys [video sound image]} opts]
     (user/api-call text video)))
0.2s

Specification

(ns user)
;; Specification
(require '[clojure.spec.alpha :as s])
(require '[clojure.spec.test.alpha :as stest])
(s/fdef send-post
    :args (s/cat :text ::valid-text
                 :variadic (s/* any?)))
(stest/instrument `send-post)
(str "Setup Specification #1 for " *ns*)
0.3s
(s/def ::valid-text (s/and string? #(<= (count %) 280)))
0.1s

Call

(send-post "lorem ipsum" :video "xyz.mov")
0.1s
(try 
  (send-post "Voluptatem aspernatur sit aliquid. Officiis quo ut impedit hic explicabo et laudantium. Ut eos expedita est aut iure at.Voluptatem corporis et laborum totam incidunt earum et. Aut ut nemo nisi esse aliquam est error. Et inventore sunt itaque dignissimos. Voluptas natus quam reprehenderit iure consectetur nulla. Laudantium molestias eos delectus. Odit quia nesciunt consequatur quia et omnis dolores. Voluptas accusamus occaecati ex quo porro. Dolor consequatur alias m." :video "xyz.mov")
  (catch Exception e (str "ERROR: " (.getMessage e))))
0.2s

Environment #2

(ns wordpress)
;; Specification
(require '[clojure.spec.alpha :as s])
(require '[clojure.spec.test.alpha :as stest])
(s/fdef send-post
    :args (s/cat :text ::valid-text
                 :variadic (s/* any?)))
(stest/instrument `send-post)
(str "Setup Specification #2 for " *ns*)
0.2s
(ns wordpress)
(defn send-post [text & opts] 
   (let [{:keys [video sound image]} opts]
     (user/api-call text video)))
(s/def ::valid-text (s/and string? #(> (count %) 0)))
0.2s

Call

(ns wordpress)
(try 
  (send-post "Voluptatem aspernatur sit aliquid. Officiis quo ut impedit hic explicabo et laudantium. Ut eos expedita est aut iure at.Voluptatem corporis et laborum totam incidunt earum et. Aut ut nemo nisi esse aliquam est error. Et inventore sunt itaque dignissimos. Voluptas natus quam reprehenderit iure consectetur nulla. Laudantium molestias eos delectus. Odit quia nesciunt consequatur quia et omnis dolores. Voluptas accusamus occaecati ex quo porro. Dolor consequatur alias m." :video "xyz.mov")
  (catch Exception e (str "Catch exception" (.getMessage e))))
0.1s

The the function's behavior reveals something about its environment.

Declarative Statements

I want to express my feelings rather than illustrate them.

~ Jackson Pollock

Jackson Pollock

Process

  • Artists follow intuition

  • Intuition is not always logical or sequential

  • Results can be analyzed

Analysis

Fractals: A Resonance between Art and Nature Richard Taylor, Ben Newell, Branka Spehar and Colin Clifford (December 2004)
Jackson Pollock Number 32 (1950) [D = 1.6]
Spot the Pollock by Richard Taylor

Process

  • Declarative statements are better for exploring

  • Imperative approaches are taken arbitrarily early on

The Execution Environment

Computer as a Communication Device

Tim Berners-Lee's original World Wide Web browser via http://info.cern.ch - home of the first website

Objective-C/NeXTSTEP

The 38 objects in the NextStep Application Kit. These objects provide the core functions that an application needs to run. (1989)

  • Where is the object state?

  • Where are the messages?

Objective-C → Swift

SwiftUI uses a declarative syntax so you can simply state what your user interface should do. For example, you can write that you want a list of items consisting of text fields, then describe alignment, font, and color for each field. Your code is simpler and easier to read than ever before, saving you time and maintenance. inline_formula not implemented

import SwiftUI
struct Content : View {
  
  @State var model = Themes.listModel
  
  var body: some View {
  	List(model.items, action: model.selectItem) { item in 
      Image(item.image)
      VStack(alignment: .leading) {
          Text(item.title)
  				Text(item.subtitle)
  					.color(.gray)
  }}}}
Swift

Describing the Environment

O̶b̶j̶e̶c̶t̶s̶ Data

(dissoc @re-frame.db/app-db :auth-token)
(count (-> (get-in @re-frame.db/app-db [:article :nodes]) vals))

M̶e̶t̶h̶o̶d̶s̶ Pure Functions

(let [id "578ef919-75a4-473f-bf52-5c2fdde45af4"]
	(re-frame.core/dispatch [:insert-code-node id]))

Code

How does :insert-code-node work? Nextjournal in Nextjournal details how to use grep to scan 80,000+ lines of code, grab the function, and run it.

The :insert-code-node dispatch uses seven functions from the namespace com.nextjournal.editor.operations.tree, included here:

(ns tree)
(defn root-id [db]
  (get-in db [:article :root]))
(defn root? [db node-id]
  (= node-id (root-id db)))
(defn parent-by-id [db node-id content-key]
  (->> db
       :article
       :nodes
       vals
       (filter #(some #{node-id} (content-key %)))
first))
(defn full-position-by-id
  "Returns a vector describing the position of the node in the form
  [parent-id content-key position]
  E.g
  [\"abc-123\" :nodes 2]
  Returns nil if node-id is the root node"
  [db node-id]
  (when-not (root? db node-id)
    (let [[parent content-key] (some (fn [key]
                                       (when-let [p (parent-by-id db node-id key)]
                                         [p key]))
                                     [:content :sections :nodes :inlines])
          content-pos (.indexOf (vec (content-key parent)) node-id)]
      (assert parent)
      (assert (<= 0 content-pos))
[(:id parent) content-key content-pos])))
(defn split-at-item [v i]
  (split-with #(not= % i) v))
(defn flat
  ([db] (assert db) (flat (get-in db [:article :nodes]) (get-in db [:article :root])))
  ([all-nodes current-node-id]
   (assert all-nodes)
   (let [node (all-nodes current-node-id)]
     (concat [current-node-id]
             (:content node)
(mapcat (partial flat all-nodes) (:sections node))))))
(defn closest-code-cell
  "Given an `article` and an absolute position `pos`, tries to find a code cell
   above the given position first. If there's no matching cell found above,
   it returns the closes below, nil otherwise. Accepts an optional predicate
   which all code cells considered should satisfy."
  ([article pos]
   (closest-code-cell article pos (constantly true)))
  ([article [section-id content-key idx :as pos] pred]
   (let [db {:article article}
         contents (vec (get-in article [:nodes section-id content-key]))
         node-at-pos (if (contains? contents idx)
                       (get contents idx)
                       (get-in article [:nodes section-id :sections 0]))
         flat (flat db)
         [before after] (split-at-item flat node-at-pos)
         closest-nodes (concat (reverse before) after)]
     (some
      (fn [id] (let [node (get-in article [:nodes id])]
                 (when (and (pred node) (= "code" (:kind node)))
                   node)))
closest-nodes))))
"[com.nextjournal.editor.operations.tree :as tree]"

Exploration

The :insert-code-node dispatch is just an anonymous function and can be run simply by passing in the re-frame database and a valid node-id for modification. It will return the dispatch, which is just data.

((fn [{:keys [db] :as ctx} [_ node-id attrs]]
   (let [db (dissoc db :auth-token)
         [p o s :as pos] (full-position-by-id db node-id)
         find-closest-lang (comp :language
                                 (partial tree/closest-code-cell (:article db)))
         attrs (update attrs :language #(or % (find-closest-lang pos) "julia"))]
     {:db db :dispatch {:insert-code-node-into p o s "code" attrs}}))
  {:db @re-frame.db/app-db} ["" "578ef919-75a4-473f-bf52-5c2fdde45af4"])

Call

Now that the dispatch better understood, call send-post within the context of a new environment. In this case, a Nextjournal article. Running the cell will modify the article itself.

(defn api-call [text id kind]
  (re-frame.core/dispatch [:insert-after id kind {:content text}]))
(defn send-post [text & opts] 
   (let [{:keys [video sound image id kind]} opts]
     (api-call text id kind)))
(send-post "lorem ipsum" :id "d404003e-ea11-41df-87c5-d21ef9392ba1" :kind "paragraph")

lorem ipsum

lorem ipsum

Referential Transparency

Actual space is intrinsically more powerful and specific than paint on a flat surface.

~ Donald Judd

Donald Judd

The painted depiction is an object that points to the environment. It is neither specific enough to encapsulate the environment nor is it detailed enough to embody the environment.

  • Data makes no attempt to be complete.

  • Selecting data is an act of curation.

  • When and how a simple function is invoked reveals complex choices.

Thank You

Say What You Mean: Exploring Declarative Computation in Art

@dschmudde . http://schmud.de

https://nextjournal.com/schmudde/

Appendix

{:deps
 {org.clojure/clojure {:mvn/version "1.10.0"}
  org.clojure/tools.deps.alpha
  {:git/url "https://github.com/clojure/tools.deps.alpha.git"
   :sha "f6c080bd0049211021ea59e516d1785b08302515"}}}
Extensible Data Notation

inline_formula not implementedHuman scale: existing within a duration and dimension that is within cognitive understanding

inline_formula not implementedMondrian was inspired by Paul Cézanne’s method of breaking down compositional elements into facets of color.

inline_formula not implementedhttps://developer.apple.com/xcode/swiftui/ (last reference: 14/July/2019)

Runtimes (3)