A Bitemporal tale – History. Of histories.

(adapted from https://juxt.pro/blog/posts/a-bitemporal-tale.html)

Remix (fork) this in Nextjournal to get started with your own Crux notebook.

Introduction

For those readers whose learning ability is better when affections are in place, we’re offering a short story writing experience through database transactions and queries.

Assuming you have some basic knowledge of Clojure, all you need for this tale is to add Crux to your deps which Nextjournal makes very simple. For more configuration details see here.

{:deps
 {org.clojure/clojure {:mvn/version "1.10.0"}
  org.clojure/tools.deps.alpha
  {:git/url "https://github.com/clojure/tools.deps.alpha.git"
   :sha "f6c080bd0049211021ea59e516d1785b08302515"}
  juxt/crux-core {:mvn/version "RELEASE"}}} ; "RELEASE" is the latest non-snapshot version from Clojars

deps.edn

(require '[crux.api :as crux])

12.1s

Clojure

Create a Crux node

(def node
  (crux/start-standalone-node
    {:kv-backend "crux.kv.memdb.MemKv"
     :db-dir "data/db-dir-1"
     :event-log-dir "data/event-log-dir-1"}))

; alternatively, you can go with RocksDB for a persistent storage
(comment
  juxt/crux-rocksdb {:mvn/version "RELEASE"} ; add this to your deps
  ; define node as follows
  (def node
    (crux/start-standalone-node ; it has clustering out-of-the-box though
      {:kv-backend "crux.kv.rocksdb.RocksKv"
       :db-dir "data/db-dir-1"})))

6.1s

Clojure

Letting data in

The year is 1740. We want to transact in our first character – Charles. Charles is a shopkeeper who possesses a truly magical artefact: A Rather Cozy Mug, which he uses in some of the most sacred morning rituals of caffeinated beverages consumption.

(crux/submit-tx
  node
    ; tx type
  [[:crux.tx/put

    {:crux.db/id :ids.people/Charles
     :person/name "Charles"
     ; age 40 at 1740
     :person/born #inst "1700-05-18"
     :person/location :ids.places/rarities-shop
     :person/str  40
     :person/int  40
     :person/dex  40
     :person/hp   40
     :person/gold 10000}

    #inst "1700-05-18" ; valid time (optional)
    ]])

0.5s

Clojure

Map {:crux.tx/tx-id: 1600811254349825, :crux.tx/tx-time: function Date() { [native code] }[Tue Jul 16 2019 15:50:40 GMT+0000 (UTC)]}

Ingest the remaining part of the set

0.3s

Clojure

(crux/submit-tx
  node
  [; rest of characters
   [:crux.tx/put
    {:crux.db/id :ids.people/Mary
     :person/name "Mary"
     ; age  30
     :person/born #inst "1710-05-18"
     :person/location :ids.places/carribean
     :person/str  40
     :person/int  50
     :person/dex  50
     :person/hp   50}
    #inst "1710-05-18"]
   [:crux.tx/put
    {:crux.db/id :ids.people/Joe
     :person/name "Joe"
     ; age  25
     :person/born #inst "1715-05-18"
     :person/location :ids.places/city
     :person/str  39
     :person/int  40
     :person/dex  60
     :person/hp   60
     :person/gold 70}
    #inst "1715-05-18"]])
; yields tx-data, omitted

(crux/submit-tx
  node
  [; artefacts
   ; In our tale there is a Cozy Mug...
   [:crux.tx/put
    {:crux.db/id :ids.artefacts/cozy-mug
     :artefact/title "A Rather Cozy Mug"
     :artefact.perks/int 3}
    #inst "1625-05-18"]

   ; ...some regular magic beans...
   [:crux.tx/put
    {:crux.db/id :ids.artefacts/forbidden-beans
     :artefact/title "Magic beans"
     :artefact.perks/int 30
     :artefact.perks/hp -20}

    #inst "1500-05-18"]
   ; ...a used pirate sword...
   [:crux.tx/put
    {:crux.db/id :ids.artefacts/pirate-sword
     :artefact/title "A used sword"}
    #inst "1710-05-18"]
   ; ...a flintlock pistol...
   [:crux.tx/put
    {:crux.db/id :ids.artefacts/flintlock-pistol
     :artefact/title "Flintlock pistol"}
    #inst "1710-05-18"]
   ; ...a mysterious key...
   [:crux.tx/put
    {:crux.db/id :ids.artefacts/unknown-key
     :artefact/title "Key from an unknown door"}
    #inst "1700-05-18"]
   ; ...and a personal computing device from the wrong century.
   [:crux.tx/put
    {:crux.db/id :ids.artefacts/laptop
     :artefact/title "A Tell DPS Laptop (what?)"}
    #inst "2016-05-18"]])
; yields tx-data, omitted

; places
(crux/submit-tx
  node
  [[:crux.tx/put
    {:crux.db/id :ids.places/continent
     :place/title "Ah The Continent"}
    #inst "1000-01-01"]
   [:crux.tx/put
    {:crux.db/id :ids.places/carribean
     :place/title "Ah The Good Ol Carribean Sea"
     :place/location :ids.places/carribean}
    #inst "1000-01-01"]
   [:crux.tx/put
    {:crux.db/id :ids.places/coconut-island
     :place/title "Coconut Island"
     :place/location :ids.places/carribean}
    #inst "1000-01-01"]])

0.3s

Clojure

Map {:crux.tx/tx-id: 1600811254775809, :crux.tx/tx-time: function Date() { [native code] }[Tue Jul 16 2019 15:50:40 GMT+0000 (UTC)]}

Looking Around : Basic Queries

Get a database value and read from it consistently. Crux uses datalog query language. I’ll try to explain the required minimum, and I recommend learndatalogtoday.org as a follow up read.

(def db (crux/db node))

; we can query entities by id
(pprint (crux/entity db :ids.people/Charles))

0.8s

Clojure

; Datalog syntax : query ids
(crux/q db
        '[:find ?entity-id ; datalog's find is like SELECT in SQL
          :where
          ; datalog's where is quite different though
          ; datalog's where block combines binding of fields you want with filtering expressions
          ; where-expressions are organised in triplets / quadruplets

          [?entity-id    ; first  : usually an entity-id
           :person/name  ; second : attribute-id by which we filter OR which we want to pull out in 'find'
           "Charles"  ; third  : here it's the attribute's value by which we filter
           ]])

0.3s

Clojure

Set(1) #{Vector(1)}

; Query more fields
(pprint
 (crux/q db
        '[:find ?e ?name ?int
          :where
          ; where can have an arbitrary number of triplets
          [?e :person/name "Charles"]

          [?e :person/name ?name]
          ; see – now we're pulling out person's name into find expression

          [?e :person/int  ?int]]))

0.7s

Clojure

(crux/q (crux/db node)
        '[:find ?name
          :where
          [_ :artefact/title ?name]])

0.4s

Clojure

Undoing the Oopsies : Delete and Evict

Ok yes, magic beans once were in the realm, and we want to remember that, but following advice from our publisher we’ve decided to remove them from the story for now. Charles won’t know that they ever existed!

(pprint
 (crux/submit-tx
  node
  [[:crux.tx/delete :ids.artefacts/forbidden-beans
    #inst "1690-05-18"]]))

0.6s

Clojure

Sometimes people enter data which just doesn’t belong there or that they no longer have a legal right to store (GDPR, I’m looking at you). In our case, it’s the laptop, which ruins the story consistency. Lets completely wipe all traces of that laptop from the timelines.

(pprint
 (crux/submit-tx
  node
  [[:crux.tx/evict :ids.artefacts/laptop]]))

0.5s

Clojure

Let’s see what we got now

(crux/q (crux/db node)
        '[:find ?name
          :where
          [_ :artefact/title ?name]])

0.1s

Clojure

Set(4) #{Vector(1), Vector(1), Vector(1), Vector(1)}

; Historians will know about the beans though
(def world-in-1599 (crux/db node #inst "1599-01-01"))
(crux/q world-in-1599
        '[:find ?name
          :where
          [_ :artefact/title ?name]])

0.1s

Clojure

Set(1) #{Vector(1)}

Plot Development : DB References

Let’s see how Crux handles references. Give our characters some artefacts. We will do with function as we will need it later again.

(defn first-ownership-tx []
  [; Charles was 25 when he found the Cozy Mug
   (let [charles (crux/entity (crux/db node #inst "1725-05-17") :ids.people/Charles)]
     [:crux.tx/put
      (update charles
              ; Crux is schemaless, so we can use :person/has however we like
              :person/has
              (comp set conj)
              ; ...such as storing a set of references to other entity ids
              :ids.artefacts/cozy-mug
              :ids.artefacts/unknown-key)
      #inst "1725-05-18"])
    ; And Mary has owned the pirate sword and flintlock pistol for a long time
    (let [mary  (crux/entity (crux/db node #inst "1715-05-17") :ids.people/Mary)]
      [:crux.tx/put
       (update mary
              :person/has
              (comp set conj)
              :ids.artefacts/pirate-sword
              :ids.artefacts/flintlock-pistol)
       #inst "1715-05-18"])])

(def first-ownership-tx-response
  (crux/submit-tx node (first-ownership-tx)))

0.3s

Clojure

user/first-ownership-tx-response

Note that transactions in Crux will rewrite the whole entity, there are no partial updates and no intentions to put them in the core as of yet. This is because the core of Crux is intentionally slim, and features like partial updates shall be kept in the upcoming convenience projects!

Who Has What : Basic Joins

(def who-has-what-query
  '[:find ?name ?atitle
    :where
    [?p :person/name ?name]
    [?p :person/has ?artefact-id]
    [?artefact-id :artefact/title ?atitle]])

(crux/q (crux/db node #inst "1726-05-01") who-has-what-query)
; yields
#{["Mary" "A used sword"]
  ["Mary" "Flintlock pistol"]
  ["Charles" "A Rather Cozy Mug"]
  ["Charles" "Key from an unknown door"]}

(pprint (crux/q (crux/db node #inst "1716-05-01") who-has-what-query))

0.7s

Clojure

A few convenience functions

0.6s

Clojure

(defn entity-update
  [entity-id new-attrs valid-time]
  (let [entity-prev-value (crux/entity (crux/db node) entity-id)]
    (crux/submit-tx node
      [[:crux.tx/put
        (merge entity-prev-value new-attrs)
        valid-time]])))

(defn q
  [query]
  (crux/q (crux/db node) query))

(defn entity
  [entity-id]
  (crux/entity (crux/db node) entity-id))

(defn entity-at
  [entity-id valid-time]
  (crux/entity (crux/db node valid-time) entity-id))

(defn entity-with-adjacent
  [entity-id keys-to-pull]
  (let [db (crux/db node)
        ids->entities
        (fn [ids]
          (cond-> (map #(crux/entity db %) ids)
            (set? ids) set
            (vector? ids) vec))]
    (reduce
      (fn [e adj-k]
        (let [v (get e adj-k)]
          (assoc e adj-k
                 (cond
                   (keyword? v) (crux/entity db v)
                   (or (set? v)
                       (vector? v)) (ids->entities v)
                   :else v))))
      (crux/entity db entity-id)
      keys-to-pull)))


; Charles became more studious as he entered his thirties
(entity-update :ids.people/Charles
  {:person/int  50}
  #inst "1730-05-18")

; Check our update
(pprint (entity :ids.people/Charles))

0.6s

Clojure

; Pull out everything we know about Charles and the items he has
(entity-with-adjacent :ids.people/Charles [:person/has])

0.0s

Clojure

Map {:person/str: 40, :person/dex: 40, :person/has: Set(2), :person/location: :ids.places/rarities-shop, :person/hp: 40, :person/int: 50, :person/name: "Charles", :crux.db/id: :ids.people/Charles, :person/gold: 10000, :person/born: function Date() { [native code] }[Tue May 18 1700 00:00:00 GMT+0000 (UTC)]}

What Was Supposed To Be The Finale

Mary steals The Mug in June

(let [theft-date #inst "1740-06-18"]
  (crux/submit-tx
    node
    [[:crux.tx/put
      (update (entity-at :ids.people/Charles theft-date)
              :person/has
              disj
              :ids.artefacts/cozy-mug)
      theft-date]
     [:crux.tx/put
      (update (entity-at :ids.people/Mary theft-date)
              :person/has
              (comp set conj)
              :ids.artefacts/cozy-mug)
      theft-date]]))

(pprint (crux/q
 (crux/db node #inst "1740-06-18")
 who-has-what-query))

0.7s

Clojure

So, for now, we think we’re done with the story. We have a picture and we’re all perfectly ready to blame Mary for stealing a person’s beloved mug. Suddenly a revelation occurs when an upstream data source kicks in. We uncover a previously unknown piece of history. It turns out the mug was Mary’s family heirloom all along!

Correct The Past

(let [marys-birth-inst #inst "1710-05-18"
      db        (crux/db node marys-birth-inst)
      baby-mary (crux/entity db :ids.people/Mary)]
  (crux/submit-tx
    node
    [[:crux.tx/cas
      baby-mary
      (update baby-mary :person/has (comp set conj) :ids.artefacts/cozy-mug)
      marys-birth-inst]]))

; ...and she lost it in 1723
(let [mug-lost-date  #inst "1723-01-09"
      db        (crux/db node mug-lost-date)
      mary      (crux/entity db :ids.people/Mary)]
  (crux/submit-tx
    node
    [[:crux.tx/cas
      mary
      (update mary :person/has (comp set disj) :ids.artefacts/cozy-mug)
      mug-lost-date]]))

(pprint
 (crux/q
  (crux/db node #inst "1715-05-18")
  who-has-what-query))

0.6s

Clojure

; Ah she doesn't have The Mug still.
; Because we store that data in the entity itself
; we now should rewrite its state on "1715-05-18"

(crux/submit-tx node (first-ownership-tx))

(pprint
 (crux/q
  (crux/db node #inst "1715-05-18")
  who-has-what-query))

0.6s

Clojure

Ah, much better!

Note that with this particular data model we should also rewrite all the artefacts transactions since 1715. But since it matches the tale we can omit the labour for this time. And if acts of ownership were separate documents, the labour wouldn’t be needed at all.

Fin

(pprint
 (crux/q
  (crux/db node #inst "1740-06-19")
  who-has-what-query))

0.6s

Clojure

Now, knowing the corrected picture we are more ambiguous in our rooting for Charles or Mary.

Also we are still able to see how wrong we were as we can rewind not only the tale’s history but also the history of our edits to it. Just use the tx-time of the first ownership response.

(pprint
 (crux/q
  (crux/db node
           #inst "1715-06-19"
           (:crux.tx/tx-time first-ownership-tx-response))
  who-has-what-query))

0.6s

Clojure

0.1s

Clojure

What’s Next?

Crux has many other important features which we left out of scope of this tale, including:

A first-class Java API, so you can use it from Kotlin, Java, Scala or any other JVM-hosted language.

There’s history API, so for every entity you get its full history, or history bounded by valid time and transaction time coordinates.

Clustering

Datalog Rules – powerful query extensions

Evict can be scoped

A Bitemporal tale – History. Of histories.

Introduction

Create a Crux node

Letting data in

Looking Around : Basic Queries

Undoing the Oopsies : Delete and Evict

Plot Development : DB References

Who Has What : Basic Joins

A few convenience functions

What Was Supposed To Be The Finale

Correct The Past

Fin

What’s Next?

Learn more at https://juxt.pro/crux