Lambda Island / Feb 04 2020

Pitch: Kaocha autotest

This document follows the structure outlined in Shape Up, chapter 6: write the Pitch.

Problem

On big test suites the current behavior of --watch still does not provide good enough feedback cycles, because it re-runs the whole test suite. We want to provide a smarter --watch version (tentatively called autotest) which uses namespace dependency information to run a more focused subset of tests after a change.

Appetite

We are not using the actual Shape Up cycles, so we use this section differently. The appetite is about drawing the line, about enabling developers to decide when to cut or hammer scope. It is about finding the balance between time invested and returned value.

This is a medium sized feature. We will implement this feature when there is €512 in budget available that can be allocated to it.

Background

Currently --watch mode runs the complete set of tests on every change, except if any tests failed in the previous run, in that case the failed tests are run first, and if that run passes (i.e. the failed tests are fixed), then do we re-run all tests.

Solution

This proposal is to add a new --watch mode which is more conservative: it uses namespace dependency information to only re-run test namespaces that are affected by a certain change.

Prior art

It seems midje has a feature like this, called "autotest". An interesting extra it provides is debouncing, by default changes are only detected once every two seconds. This can prove useful when saving multiple files at once (e.g. Emacs save-some-buffers, C-x s)

Design choices

Implementing this for clojure.test based tests is straightforward, since there a namespace corresponds with a test type. However the mechanism should not be clojure.test specific, instead being extensible to new test types, so that ClojureScript, Cucumber or other tests can also benefit from it. There are three cases to consider

ClojureScript: we have to extend the clojure.tools.namespace dependency analysis to also happen for ClojureScript. Test types can reuse the existing :kaocha.ns/name key on testables, we can add an extra :kaocha.ns/type key to distinguish clj vs cljs, since these will require separate dependency graphs.
Non clojure.test testing libraries for Clojure: here we can still rely on the dependency graph information as used for clojure.test, assuming test types add :kaocha.ns/name on their testables.
Test types that have no clear correspondence with source files, like cucumber. For these we can introduce a metadata convention: ^:kaocha/test-subject ns-sym(or even var-sym). This could also be useful in clojure/cljs based tests to narrow down what tests to re-run, and for other future use cases like mutation testing.

What do we do we do on startup? There is no reload information at that point, so either we immediately run the full suite (this is what --watch does), or we simply wait for the first change. I think in this case waiting is the better option.

How to invoke this new mode? Here I'm not sure and further brain storming might be needed, as I'm not sure what the best name is for this feature. --autotest might do the trick, although it will be confusing to have both --watch and --autotest features. Or it could be an option that changes the behavior of --watch.