Pitch: kaocha-cljs controller approach (WIP)

This document follows the structure outlined in Shape Up, chapter 6: write the Pitch.

Problem

Currently kaocha-cljs is backed by a ClojureScript repl-env. This has served us well, but in the past year we've discovered several shortcomings, which we want to address.

  • The repl-env implementations provide very little debugging information, when something goes wrong it's very hard to troubleshoot.

  • We are unable to service people who can't get a conformant repl-env implementation for their system. Shadow-cljs in particular does not implement the standard ClojureScript repl-env.

  • repl-env's are tied up with specific compilation strategies and settings, if you want to use specific compiler options this will likely break the repl.

  • You can't test optimized (whitespace/simple/advanced) builds

A separate issue but somewhat related is that we don't re-use repl-env's between runs, so in a REPL context when you start Kaocha ten times using a browser/repl-env, you get ten now browser tabs.

Appetite

We are not using the actual Shape Up cycles, so we use this section differently. The appetite is about drawing the line, about enabling developers to decide when to cut or hammer scope. It is about finding the balance between time invested and returned value.

Background

The traditional approach taken by ClojureScript testing tools like doo is to compile all tests to a single blob with a single entry points, which contains the tests as well as the test runner. At that point your only option is to run this blob and wait until it finished.

For kaocha-cljs we wanted to move away from this and towards a more dynamic system that integrates with the existing kaocha workflow and tooling. In this model Kaocha (running on the Clojure side) is in charge. It builds up a test plan based on the test configuration and user input, then loads and runs those tests dynamically. Plugins can hook into various parts of this process. We want to be able to re-use these plugins as well as the existing reporters (the component providing test output).

The solution has been to use a cljs (p)repl-env, like cljs.node/repl-env, or cljs.browser/repl-env. These allow us to invoke tests one by one by sending forms to the REPL. cljs.test events are collected and sent back to Kaocha via a websocket, and routed to the reporter.

The build up the test plan to know what to test we invoke the cljs analyzer to find the test vars that are present (we need var metadata for this).

This has allowed us to reuse things like the junit-xml plugin, the watch feature, metadata filtering, reporter implementation like dots/progress bar/documentation, and more. It also means we don't need to precompile anything, since we require the necessary namespaces on the fly, and we can decide to skip tests at the last moment, e.g. when using the --fail-fast feature, or when having custom filtering logic in a hook or plugin.

So this approach has bought us a lot, but we have become more aware of the shortcomings, as listed up above. These essentially stem from the fact that the repl-env abstractions complects compilation, and managing a connection with a JavaScript environment.

Solution

Load step

What Kaocha calls the load step is really about taking a test configuration (test type, test directories, and test-type specific settings), a current selection (based on test ids or metadata), and convert it to a test plan, a hierarchical structure of tests to execute. We do this by ananlyzing cljs var metadata.

Then we invoke the compilation, injecting a generated :main namespace. This namespace requires all the test namespaces (that are in the test plan and are not already marked as skipped), as well a mapping from test id to test function (so we can dynamically invoke them), and finally it requires the "test controller", a new component that coordinates with Kaocha (on the Clojure side, the "server" in a sense).

Finally we load this compiled blob into a JavaScript environment, and wait for it to connect back to us. Or if we already have an existing connection, we instruct it to load the new definitions.

The end result is that we have a JavaScript environment waiting over a websocket for instructions on what to do.

Run step

Kaocha now starts doing its thing by walking the graph of testables, and invoking their testable/-run implementations. For ClojureScript test vars this will mean sending a signal over the websocket to the controller, telling it to run a certain test, and then waiting for a signal back that the test has finished (or time out).

Meanwhile cljs.test events come back over the websocket and get routed to the reporter (test started, assertion failed, etc.), this part we already do today.

Stateful REPL use

Currently a new kaocha run will always start a new cljs environment (e.g. open a new browser tab, start a new node process). We want to be able to reuse an existing environment though (especially important in case of browser use). This has two implications

  • kaocha-cljs will no longer be stateful, we will need to cache connections for later use

  • we need to be able to load new test definitions into an existing runtime. Note that since we want to support arbitrary compilation modes this is different from what figwheel (and I believe shadow) do, as though hot-reload modes work on a per-namespace level and are not compatible with Google Closure optimizations. Instead we need to load the new compiled blob, and make sure it replaces what is there or does adequate name munging to prevent collisions.

Rabbit Holes

  • Details about the solution worth calling out to avoid problems

  • Areas of uncertainty that need to be evaluated first to make sure they are not unacceptable time sinks

No-goes

Anything specifically excluded from the concept: functionality or use cases we intentionally aren’t covering to fit the appetite or make the problem tractable.

Resources / Links

Links to related Github issues, notebooks, documentation, etc.