Howto: Reproducible Benchmarks

It's sometimes not that simple to get some benchmark running, especially if it depends on lots of software! It's even worse when a benchmarks runs, but one forgets to set e.g. an environment variable that would speed up the baseline, so you euphorically publish a blog post about how you beat Benchmark X - but you simply failed to correctly reproduce the benchmark. To save people from such embarassements, this article will give some tips on how to publish your benchmarks.

Luckily, nextjournal almost by coincident came up with the perfect platform to share your benchmarks and enable readers to immediately run and reproduce them! Nextjournal is a platform where you can publish articles that contain code and data, all frozen into a runnable docker container! This is supposed to make research more reproducable but can also be used for benchmarking. The beauty is, that every article can be "remixed": that means you can make a new version of an existing article in the exact same environment (running in the cloud) as the original author. So if you find a benchmarking article hosted on nextjournal, you can easily extend it with your own version. You can also combine the environments of multiple articles into one, making it easy to compare benchmarks written in different languages or frameworks.

In this article, I will write my own benchmarks and analysis, using and explaining the tools nextjournal offers us! Ironically, it was quite a bit of work to set up and measure the C benchmarks used in this article correctly - work one really doesn't want to do when quickly trying to get a baseline on the own hardware!

When benchmarking, the first goal is to create statistical significant measurments and comparisons - which turns out to be quite the science in itself. Even when running the same code on the same hardware, there is enough noise on most machines to spoil the measurements. Let's first explore how bad noise can be for our measurements! With nextjournal you have the option to use a "shared" runner or a "dedicated" runner to execute your code. Shared ones will share the CPU with multiple other users, while a dedicated runner will boot up a whole machine for your article. As you can already guess, a shared runner probably has much more noise - but how bad can it be and how reliable is a dedicated machine? I prepared two articles running benchmarks from the BenchmarkGame - which I remixed and run on dedicated and shared hardware.

The benchmarks are run using BenchmarkTools, a framework that applies lots of tricks to eliminate noise. BenchmarkTools times the same code until a significant number of samples is reached, which then gets encapsulated in a single benchmarking object. Those objects can then be compared and tested if they contain a statistical significant difference. For the above articles, I record around 1000 benchmark objects for each code sample.

The most interesting statistic is: how high is the likelyhood, that I benchmark the exact same native code, but measure a significant difference in execution speed. For that, let's compare the benchmark samples generated from the same code against itself!