Pitch: test selections (draft)

This document follows the structure outlined in Shape Up, chapter 6: write the Pitch.

Problem

We have a number of ways in which users can select a subset of tests to run, but they don't fit all use cases. This leads to confused users who try to use the current features to their advantage, but don't get the results they expect.

In particular it's common to want to split up a test suite, for instance in slow and fast tests.

;; tests.edn

#kaocha/v1

{:tests [{:id :fast

          :skip-meta [:test/slow]}

         {:id :slow

          :focus-meta [:test/slow]}]} 

This kind of works, but behaves weirdly when combined with `--focus` / --skip command line flags, and it can confuse plugins, because it leads to a test plan where ids are not unique.

Appetite

• How much time we want to spend

• how that constrains the solution

Background

Running a subset of tests is done either by focusing on specific tests (a positive selection), or skipping tests (a negative selection).

Focusing or skipping can be done either on test ids (e.g: run test foo, bar, and baz), or on test metadata (e.g. run all tests that have the metadata :test/api).

These flags (focus, skip, focus-meta, skip-meta) can be set per test run on the command line, or they can be configured on a test suite in the tests.edn.

# Command line

bin/kaocha --focus foo --focus bar --focus baz

bin/kaocha --focus-meta :test/api

bin/kaocha --skip-meta :test/slow

;; tests.edn

#kaocha/v1

{:tests [{:id :unit

          :skip-meta [:test/wip]}]} ;; skip tests that are marked as "work in progress"

It is technically also possible to configure focus/skip at the top level of the test configuration, but this is an implementation detail and not documented or recommended.

The actual filtering is done by an internal plugin (it's a plugin, but it's enabled behind the scenes), by recursively going over the test plan, and marking tests as :kaocha.testable/skip. In other words, whether you use focus or skip, in the end it all gets converted to skip.

Solution

The proposal is to introduce a new concept named test selections, these are configured outside of the test suites. They have an identifier, and a combination of test ids and metadata keys.

#kaocha/v1

{:plugins [:kaocha.plugins.alpha/selections]

 :tests [...]

 :kaocha.plugins.alpha/selections

 {:api {:tests [foo.api-test bar.api-test]}

  :slow {:meta [:test/slow]}}}

These again provide a way to take a subset of the test plan. But

• they are configured in tests.edn and have a name, so they can be shared and maintained by the team

• They are outside the test suite configurations. They can be mixed and matched with test suites

• They don't specify whether it's a positive (focus) or negative (skip) selection, this is done when they are invoked

# run the slow tests in the unit test suite

bin/kaocha --select slow unit

# run the slow tests in the integration test suite

bin/kaocha --select slow integration

# run the slow tests in any test suite

bin/kaocha --select slow 

# skip the slow tests, run all non-slow tests

bin/kaocha --except slow unit

Note that --select / --except is tentative, we might want to think a bit harder about what to name these flags.

Rabbit Holes

• Details about the solution worth calling out to avoid problems

• Areas of uncertainty that need to be evaluated first to make sure they are not unacceptable time sinks

No-goes

Anything specifically excluded from the concept: functionality or use cases we intentionally aren’t covering to fit the appetite or make the problem tractable.

Resources / Links

Links to related Github issues, notebooks, documentation, etc.