Pitch: Kaocha doctests

This document follows the structure outlined in Shape Up, chapter 6: write the Pitch.

Problem

In API docs it can be very helpful to provide a few well chosen example invocations together with their results. These are essentially example based tests, but we don't treat them as such. We should have a standard way of writing these, and tooling to verify them.

(defn my-sort 

  "Sorts a sequence.

  > (my-sort [4 1 3])

  ;;=> (1 3 4)

  "

  [s]

  (sort s))

Appetite

We are not using the actual Shape Up cycles, so we use this section differently. The appetite is about drawing the line, about enabling developers to decide when to cut or hammer scope. It is about finding the balance between time invested and returned value.

This is a large feature. We will implement this feature when there is €1024 in budget available that can be allocated to it.

Background

This is a new feature, but there is some prior art. First of all Clojure already has a mechanism for attaching simple tests to a function, through metadata.

(defn my-sort

  {:test #(assert (= (my-sort [4 1 3]) [1 3 4]))}

  [s]

  (sort s))

This is rarely used in practice, because it has some real downsides. Having to use assert is awkward and leads to unhelpful output. These tests also don't show up in documentation.

In terms of prior art we should definitely mention Doctests as found in Python, and implemented in Elixir, Elm, and elsewhere.

def list_to_0_index(lst):

    """A solution to the problem given in:

    https://rgrig.blogspot.com/2005/11/writing-readable-code.html

    'Given a list, lst, say for each element the 0-index where it appears for

    the first time. So the list x = [0, 1, 4, 2, 4, 1, 0, 2] is

    transformed into y = [0, 1, 2, 3, 2, 1, 0, 3]. Notice that for all

    i we have x[y[i]] = x[i]. Use any programming language and any data

    representation you want.'

    >>> x = [0, 1, 4, 2, 4, 1, 0, 2]

    >>> list_to_0_index(x)

    [0, 1, 2, 3, 2, 1, 0, 3]

    >>>

    """

    return [lst.index(i) for i in lst]

Solution

This can be implemented as a new test type

#kaocha/v1

{:tests [{:id :doctests

          :type :kaocha.type/doctest

          :test-paths ["src"]}]}

Largely following the pattern of other test types, like kaocha-cljs or cucumber. See the docs for Extending: test-types.

Rabbit Holes

The biggest challenge here (and the biggest potential for bike shedding) is to decide on the rules for parsing. We should try to clear this out first, keeping in mind that the solution should render nicely on cljdoc, so perhaps using markdown markers is a good idea, or using 4 space indent so it's considered a code block.

No-goes

We only want to support basic call-function-and-compare-with-output, i.e. the kind of in/out you would get from a REPL. We're not going to go into test assertions, etc. The output should basically be what you get for a (is (= ...)).

Resources / Links