Pitch: Kaocha doctests
This document follows the structure outlined in Shape Up, chapter 6: write the Pitch.
In API docs it can be very helpful to provide a few well chosen example invocations together with their results. These are essentially example based tests, but we don't treat them as such. We should have a standard way of writing these, and tooling to verify them.
We are not using the actual Shape Up cycles, so we use this section differently. The appetite is about drawing the line, about enabling developers to decide when to cut or hammer scope. It is about finding the balance between time invested and returned value.
This is a large feature. We will implement this feature when there is €1024 in budget available that can be allocated to it.
This is a new feature, but there is some prior art. First of all Clojure already has a mechanism for attaching simple tests to a function, through metadata.
This is rarely used in practice, because it has some real downsides. Having to use assert is awkward and leads to unhelpful output. These tests also don't show up in documentation.
In terms of prior art we should definitely mention Doctests as found in Python, and implemented in Elixir, Elm, and elsewhere.
This can be implemented as a new test type
Largely following the pattern of other test types, like kaocha-cljs or cucumber. See the docs for Extending: test-types.
The biggest challenge here (and the biggest potential for bike shedding) is to decide on the rules for parsing. We should try to clear this out first, keeping in mind that the solution should render nicely on cljdoc, so perhaps using markdown markers is a good idea, or using 4 space indent so it's considered a code block.
We only want to support basic call-function-and-compare-with-output, i.e. the kind of in/out you would get from a REPL. We're not going to go into test assertions, etc. The output should basically be what you get for a (is (= ...)).
Resources / Links