# Reproducing "A Large-Scale Study of Programming Languages and Code Quality in GitHub: A Reproducibility Study"

<h2 class="pm-node nj-subtitle">A Reproducibility-Study Reproducibility Study</h2>

<p class="pm-node nj-authors">Emery D. Berger, Celeste Hollenbeck, Petr Maj, Olga Vitek, Jan Vitek</p>

This is an attempt to reproduce the data processing and analysis from [this paper](https://arxiv.org/abs/1901.10220), from its associated [Github repo](https://github.com/PRL-PRG/TOPLAS19_Artifact), itself a reproducibility study on an earlier paper. The initial method used to create this notebook is to process all of the `.Rmd` (R markdown) files in the repo's root directory into one file (done [here](https://nextjournal.com/mpd/toplas19-rmd-processing)), and then import that file into Nextjournal. The result will have data and library codes filled in, formatting issues corrected, and any required code changes made to get processing running and results displayed will be annotated.

# Building the Artifact of the original paper (original_artifact.Rmd)

The following command extract the contents of the original artifact into the `original` directory alongside the archive. The original artifact contains two folders, the `R_code` folder in which the original R scripts can be found as they were given to us and the `sqlDump` folder, which contains the data.

```bash id=ddd0c143-9f39-4ae2-a050-fb4d2b3a29d4
cd original
tar -zxf artifact.tgz
cd ..
```

After the original artifact is extracted, the sql data dump must be combined together to produce the two files the rest of the paper is concerned with, namely the `_everything.csv` file containing th entire dataset and th `_newSha.csv` containing filtered data. The original artifact contained the `rebuild.sh` script to produce these files, which can be found in the `original/sqlDump/DATA` directory:

```bash id=e00ac52a-7a95-4790-aaca-84de5f73d4bc
cd original/sqlDump
./rebuild.sh
cd ../..
```

# A Large Scale Study of Programming Languages and Code Quality in Github --- Repetition (repetition.Rmd)

Initially we load the implementation and initialize the environment making sure that all outputs of this script will be stored in `artifact/repetition` subfolder:

```r id=75a31db9-fcc0-4575-9b24-b32676d773a7
# load the file containing the actual implementation details
knitr::opts_chunk$set(echo = FALSE)
source("implementation.R")
initializeEnvironment("./artifact/repetition")
```

## Data collection (chapter 2.2 of the FSE manuscript)

We staert with data acquisition. The artifact obtained from the original authors contained two CSV files. The file `everything` contains one row per unique commit and filename pair. The file `newSha` contains one row per unique commit and language pair. Thus, one row of `newSha` summarizes one or more rows of `everything`. By looking at the code of the artifact we have determined that the contents of the `newSha` file was used for the analysis.

```r id=847680ae-87d8-4cf5-a51a-1a89056ab4ca
everything <- loadEverything()
newSha <- loadNewSha()
```

### Number of projects analyzed

The study claims that `729` projects were used and that each one of them had more than 27 commits in any language used:

```r id=17374c10-375e-4d3c-8ce7-2b8a6e400923
newSha_project_names = sort(unique(newSha$project))
number_of_projects <- length(newSha_project_names)
check(number_of_projects==729) 
projects_with_too_few_commits = newSha %>% group_by(project, language) %>% dplyr::summarize(n = n()) %>% filter(n <= 27)
check(nrow(projects_with_too_few_commits) == 0)
```

Let's see how many projects are there in `everything`:

```r id=687707b5-7683-4dfd-aa96-9fbd33c2b6ac
# There are projects in the larger file that are missing from the smaller file, with no explanation in the paper
everything_project_names <-  sort(unique(everything$project))
assert_that(length(everything_project_names)==877)
# some projects are not used in the paper
projects_not_in_paper    <-  setdiff(everything_project_names, newSha_project_names)
number_of_projects_not_included <- length(projects_not_in_paper)
check(number_of_projects_not_included == 148)
# some commits are not used
everything %>% filter(project %in% projects_not_in_paper) -> everything_unused
# and the rest is used
everything %>% filter( project %in% newSha_project_names)  -> everything_used
assert_that(nrow(everything_used)==4956282)
```

Ok, that's more, check if the discrepancy is explained by too few commits for the missing projects:

```r id=93cd9dff-5a48-4751-8f8c-de31c3d21654
number_of_large_projects_not_included <- nrow(everything %>% dplyr::filter(project %in% projects_not_in_paper) %>%  group_by(project,tag) %>% dplyr::summarize (n=n()) %>% filter(n>27))
check(number_of_large_projects_not_included == 101)
out("numberOfProjectsIncluded", number_of_projects )
out("numberOfProjectsNotIncluded", number_of_projects_not_included)
out("numberOfLargeProjectsNotIncluded", number_of_large_projects_not_included)
```

We found data for `148` projects that have been ignored in the study. Although the study filtered out projects with fewer than 28 commits, 101 of the ignored projects have 28 commits or more.

### Sizes of the code analyzed

The study claims that `63M` SLOC were analyzed and uses the difference betweeen insertions and deletions as reported by git to calculate this. Although this metric is not precise, we follow with it:

```r id=203f844b-df5e-41b8-b2b7-28887630762d
sloc <- sum(newSha$insertion) - sum(newSha$deletion)
sloc_mio <- round(sloc / 1000000, 1)
check(sloc_mio == 63) # what the paper claims
out("slocMio", sloc_mio)
check(sloc_mio == 80.7)
```

In fact, there are There are 80.7 mio lines of code

### Claim: 29K authors

```r id=95a93fec-3d4a-4b1c-8419-ca54e94303be
number_authors <- length(unique(everything_used$author))
check(number_authors / 1000 == 29) # what the paper says
check(number_authors == 47297)
number_committers <- length(unique(everything_used$committer))
check(number_committers == 29082)
out("numberCommitters", round(number_committers/1000, 0))
out("numberAuthors", round(number_authors/1000, 0))

everything %>% filter(committer == "Linus Torvalds") -> lt
everything %>% filter(author == "Linus Torvalds") -> lt2
out("linusCommitter", nrow(lt))
out("linusAuthor", nrow(lt2))
```

There are `29K` committers, but the study is interested in active developers and not just people with commit rights. There are `47K` developers.

### Claim:  1.5M commits

```r id=d703ad29-5f8c-4391-8965-3b8eff3d5221
number_of_commits <- length(unique(newSha$sha))
check(number_of_commits == 1478609)
out("numberCommitsMio", round(number_of_commits/1000/1000,1))
```

There are 1.5m commits.

### Claim: 17 languages

```r id=713cd3e2-0afc-474b-aeb0-4ae264260404
number_of_languages = length(unique(newSha$language))
check(number_of_languages == 17)
```

The authors have indeed operated with `17` languages.

### Project-Language pairs with fewer than 20 commits are ignored

The authors claim that they have excluded project-language pairs for which there was fewer than 20 commits in project. Let's see if there are any such pairs in `newSha`:

```r id=7bb48ab6-41c7-4963-b80f-c1542b2e2e3e
newSha %>% group_by(project, language) %>% dplyr::summarize(n=n()) %>% filter(n < 20) -> too_small_to_keep
check(nrow(too_small_to_keep) == 0)
```

There are none, so the data has already been cleaned.

### Claim: 564,625 buggy commits.

```r id=48742117-495b-40e9-8ed7-15f3f061b494
newSha %>% filter(isbug == 1) -> bugFixes
numberOfBugFixes <-  length(unique(bugFixes$sha))
check(numberOfBugFixes == 530777)
out("numberOfBugFixes", numberOfBugFixes)
```

The number of bug fixes is close but not quite matching. We have about `30K` fewer.

### everything to newSha summary

Let's now summarize `everything` which contains only the projects used in `newSha` to check for any further discrepancies wrt `newSha`. First we only select the columns we will need and convert discrete data into factors to limit memory footprint and speedup execution:

```r id=3b0bfb28-8ae8-49e3-9075-bba71323c03d
# The data frames have redundancy. Delete columns we can derive and remove columns we don't use. Turn character columns into factors.
everything %>% dplyr::select(tag,         # language assigned by authors
                           author,             # name of author
                           committer,          # name of committer (not same as author)
                           commit_date,        # when was the commit made
                           commit_age,         # ???
                           insertion,          # number of lines inserted
                           deletion,           # number of lines deleted
                           sha,                # commit id
                           project,            # project name
                           file_name,           # commit filename
                           domain,
                           isbug
                          )  -> everything_clean

# turn discrete data into factors  
everything_clean %>% mutate(language = as.factor(tag),
                               committer = as.factor(committer),
                               project = as.factor(project)
                               ) -> everything_clean
```

Next, there are some redundancies in `everything`, such as rows that differ in `github_languag` column, but describe the same file (we found `23210` such rows). Since we are not interested in the `github_language` column any more, we can simply ignore such redundancies:

```r id=3a3392d6-c871-4e32-bbed-374ada0ac49b
# We remove keep only one row per unique columns we are interested in
everything_clean %>% distinct(sha,file_name, project,.keep_all=TRUE)  -> everything_clean
```

To ease our life down the road, we also rename the languages factor to use same names as does the `newSha` version, and order the factor in alphabetical order for better visualization:

```r id=6d6c87a4-c7e9-4fe8-b8b0-d9739c28ca1a
# sanity: use the same labels as in newSha
everything_clean$language <-  
  revalue(everything_clean$language, c( "c"="C",
                                        "clojure"="Clojure",
                                        "coffeescript"="Coffeescript",
                                        "cpp"="C++","csharp"="C#",
                                        "erlang"="Erlang",
                                        "go"="Go",
                                        "haskell"="Haskell",
                                        "java"="Java",
                                        "javascript"="Javascript",
                                        "objc"="Objective-C",
                                        "perl"="Perl",
                                        "php"="Php",
                                        "python"="Python",
                                        "ruby"="Ruby",
                                        "scala"="Scala",
                                        "typescript"="Typescript"))

# sanity: levels in alphabetical order
everything_clean$language <- 
    factor(everything_clean$language, levels=sort(levels(everything_clean$language)))
```

Finally, let's summarize `everything_clean` per language:

```r id=c2ad7b4e-cd2b-47fb-85bb-f94265699a91
checkSha = everything_clean %>% 
    dplyr::select(-(file_name)) %>% 
    group_by(sha) %>% 
    mutate(files = n()) %>% 
    distinct(sha, .keep_all = T) %>% 
    dplyr::select(language, project, sha, files, committer, author, commit_age, insertion, deletion, isbug, domain)
```

Let's see the number of unique commits found in `newSha` and the version constructed from `everything`:

```r id=363e8245-f397-4741-8cc7-29d312c72500
newSha_distinct = unique(newSha$sha) 
checkSha_distinct = unique(checkSha$sha) 
in_both = intersect(newSha_distinct, checkSha_distinct)
out("notInNewShaRaw", length(checkSha_distinct) - length(in_both))
out("notInEverythingRaw", length(newSha_distinct) - length(in_both))
```

This is promising, what we can do now is basically use `author` field from checked and append it to `newSha` which lacks the author (the original paper used committers instead as we discovered above).

```r id=06ea26de-f567-4f31-ad0a-4bf50fdedb84
stuff = checkSha %>% dplyr::select(sha, author)
newSha = inner_join(newSha, stuff, "sha")
```

Now in order to properly compare `newSha` and its version we obtained from `everything` we should remove from it any project-language pairs that have fewer than 20 commits:

```r id=a7ab23ce-4943-440a-bd06-9208cf6ffb53
checkSha = checkSha %>%
    group_by(project, language) %>%
    mutate(n = n()) %>%
    filter(n >= 20) %>%
    dplyr::select(-(n));
```

And redo the verification that it contains everything `newSha` does:

```r id=9ee9ce77-439f-4001-bfac-b83b12779b29
newSha_distinct = unique(newSha$sha) 
checkSha_distinct = unique(checkSha$sha) 
in_both = intersect(newSha_distinct, checkSha_distinct)
out("notInNewSha", length(checkSha_distinct) - length(in_both))
out("notInEverything", length(newSha_distinct) - length(in_both))
```

OUCH. So there are now commits that are in `newSha`, but are *not* in `everything` summarized. Furthermore, we should be comparing based on projects + commits since a commit may be shared with more projects:

```r id=6f6fc6a0-fbe1-4875-8577-ba3546235487
newSha_distinct = unique(paste0(newSha$project, newSha$sha))
checkSha_distinct = unique(paste0(checkSha$project, checkSha$sha)) 
in_both = intersect(newSha_distinct, checkSha_distinct)
out("notInNewShaWithProject", length(checkSha_distinct) - length(in_both))
out("notInEverythingWithProject", length(newSha_distinct) - length(in_both))
```

OUCH^2. It would seem there is *lot* of duplication in `newSha` that is not present in `everything`. Spoiler alert: More on duplication in `newSha` in the reproduction section.

As last thing in the data collection section, let us give `newSha` the same treatment we gave to `everything`, i.e. add factors, remove unused columns, etc.

```r id=c3f62e2a-a4fa-462a-b030-38a12e49067e

# newSha also has redundancies
newSha %>% dplyr::select(language,
                  typeclass,
                  langclass, 
                  memoryclass,
                  compileclass,
                  project,
                  sha,
                  files,
                  committer,
                  author,
                  commit_age,
                  commit_date,
                  insertion,
                  deletion,
                  isbug,
                  #bug_type,
                  domain,
                  btype1,
                  btype2
                ) %>%
            mutate(language = as.factor(language),
                   typeclass = as.factor(typeclass),
                   langclass = as.factor(langclass),
                   memoryclass = as.factor(memoryclass),
                   compileclass = as.factor(compileclass),
                   project = as.factor(project),
                   #committer = as.factor(committer),
                   #bug_type = as.factor(bug_type),
                   domain = as.factor(domain)
            ) -> newSha

# sanity: levels in alphabetical order
newSha$language <- 
    factor(newSha$language, levels=sort(levels(newSha$language)))
```

## Categorizing Languages (section 2.3)

While we did not have the code that categorizes the languages into groups, the classes are well described in the paper and the classification data is available in the `newSha` file. As a first step we thus verify that the categorization in `newSha` conforms to the categories explained in the paper. First, we bring the class format to single column compatible with what we use further down the study.

We reclassify `hybrid` memory model languages (Objective-C) to `Unmanaged`, which is what the paper later suggests (the claim is that while Objective-C can use both, unmanaged is the default case).

```r id=7ddec0d4-fe94-41a6-a830-363f527d3b37

newSha$typeclass = revalue(newSha$typeclass, c("strong" = "Str", "weak" = "Wea"))
newSha$langclass = revalue(newSha$langclass, c("functional" = "Fun", "proc" = "Pro", "script" = "Scr"))
newSha$memoryclass = revalue(newSha$memoryclass, c("unmanaged" = "Unm", "managed" = "Man", "hybrid" = "Unm"))
newSha$compileclass = revalue(newSha$compileclass, c("dynamic" = "Dyn", "static" = "Sta"))

newSha$combinedOriginal = paste(newSha$langclass, newSha$compileclass, newSha$typeclass, newSha$memoryclass)
```

Now, classify the languages according to the classification as described in the FSE paper:

```r id=f9524cd8-8dd1-4d91-8f68-69d366fc7d85
newSha$combined = classifyLanguageAsOriginal(newSha$language)
```

Make sure that the original and our recreated classification is identical:

```r id=07aff476-f04b-404c-8703-f40bf940cd41
check(identical(newSha$combinedOriginal, newSha$combined))
```

So no errors in this section. Let's remove `combine` column:

```r id=eadab99c-1fb7-4d9f-8fda-ea3c8f5fb16b
newSha = newSha %>% dplyr::select(-(combined))
```

Next, the paper actually ignores TypeScript from the language category analysis saying that TypeScript does not properly fit into the categories, so we assign a special category (`Other`) to it so that we can easily remove it later. Finally, we convert the `combinedOriginal` column into a factor.

```r id=c29eb259-34a0-4c9a-9450-edc64cd55e10
newSha$combinedOriginal[newSha$language == "Typescript"] = "Other"
newSha = newSha %>% mutate(combinedOriginal = as.factor(combinedOriginal))
```

### Better Language Classification

Although the language categorization in the data is same as in the paper (modulo the hybrid memory class for Objective-C and categories given for Typescript), it is wrong. We have both reclassified the obvious mistakes and marked two other languages, that do not properly fit the categories (Scala and Objective-C). Furthermore we categorize Typescript as scripting, dynamic, weak and managed, since it fully conforms to this category:

Category                          | Per Paper                      | Per Reality ----------------------------------|--------------------------------|------------- Functional-Static-Strong-Managed  | haskell scala                  | haskellFunctional-Dynamic-Strong-Managed | clojure erlang                 | (empty) Proc-Static-Strong-Managed        | java go cs                     | java go c# Proc-Static-Weak-UnManaged        | c cpp                          | C C++ Script-Dynamic-Weak-Managed       | coffee js perl php             | Python Perl Ruby JavaScript Php CoffeeScript Typescript Script-Dynamic-Strong-Managed     | python ruby                    | (empty) Functional-dynamic-weak-managed   | (empty)                        | clojure erlang Other                             | Typescript                     | Scala Objective-C

```r id=822223d7-d759-4c9b-bab4-36050bcf6421
newSha$combinedCorrected = as.factor(classifyLanguageCorrected(newSha$language))
```

## Identifying Project Domain (2.4)

The artifact available to us does not contain the code for the identification and the paper does not specify it in detail enough for us to repeat it from scratch. The data however contain the `domain` column which corresponds to the domain associated in this section. We will use this in subsequent analysis.

## Categorizing Bugs (2.5)

The artifact again did not contain any code towards this goal, nor did we find the results of such categorization anywhere in the data. We will revisit this in RQ4.

## Saving the processed data

This is not actual part of the repetition, but we save the `newSha` and `checkSha` for future use:

```r id=3728d843-30c2-44d8-8c0a-d8e687bb3758
write.csv(newSha, paste0(WORKING_DIR, "/Data/newSha.csv"))
write.csv(checkSha, paste0(WORKING_DIR, "/Data/checkSha.csv"))
```

## Are some languages more defect prone than others? (chapter 3, RQ 1)

Get the data, summarize the commits information per project & language and log transform according to the paper (in the code we have obtained log10 and log are used for different rows of the model)

```r id=8952f6fd-2180-42af-99df-263d993d405c
newSha$combined = newSha$combinedOriginal
newSha$devs = newSha$committer
X = summarizeByLanguage(newSha)
Y = logTransform(X, log10, log)
```

### Fit the Negative Binomial Regression

The Negative Binomial model is fit on the original scale of bcommits. However, the predictors were log-transformed to avoid influential observations (as was in the study). First verify the weighted contrasts to conform to the original:

```r id=c6924612-47e4-40cd-94e3-56c8f1bd6871
contr.Weights(Y$language)
```

[__out-0.csv][nextjournal#output#c6924612-47e4-40cd-94e3-56c8f1bd6871#__out-0.csv]

We see diagonal with ones and the last line (Typescript) contains the weighted contrasts, this looks good.

Let's fit the model, and then the releveled model to get the parameter for the last language:

```r id=fb2319e2-1d3e-46da-86c7-2975bb5365b0
nbfit = glm.nb(bcommits~lmax_commit_age+ltins+ldevs+lcommits+language, contrasts = list(language = contr.Weights(Y$language)), data=Y)
nbfit_r = glm.nb(bcommits~lmax_commit_age+ltins+ldevs+lcommits+language_r, contrasts = list(language_r = contr.Weights(Y$language_r)), data=Y)
# combine them into single result table
result = combineModels(nbfit, nbfit_r, Y$language)
result
```

[__out-0.csv][nextjournal#output#fb2319e2-1d3e-46da-86c7-2975bb5365b0#__out-0.csv]

Let's now juxtapose this to the FSE paper's results. The `ok` column contains `TRUE` if there was a claim in the original with a significance and we support that claim at the same significance level, `FALSE` if there was a claim and we do not support it, or support it at worse significance, and `NA` if there was no claim in the original to begin with.

```r id=1cf447fc-2b26-4153-94fd-3d5bd3fcab10
juxt = merge(result, baselineFSE_RQ1(), by = 0, all = T, sort = F)
juxt$ok = checkPValues(juxt, "FSE_pv", "pVal")
juxt
```

[__out-0.csv][nextjournal#output#1cf447fc-2b26-4153-94fd-3d5bd3fcab10#__out-0.csv]

Outside of the control variables, the only value we were not able to repeat is PHP. We believe this is a simple typo in the original paper (the authors themselves have noticed this in the CACM reprint of the paper), so we have created updated baseline:

```r id=aa6ebef6-36b2-4aa9-87fc-9bc4b5f2422d
juxt = merge(result, baselineFSE_RQ1_fixed(), by = 0, all = T, sort = F)
juxt$ok = checkPValues(juxt, "FSE_pv", "pVal")
juxt
```

[__out-0.csv][nextjournal#output#aa6ebef6-36b2-4aa9-87fc-9bc4b5f2422d#__out-0.csv]

Better. We were able to repeat the RQ1 convincingly.

### Deviance Analysis

The authors also display numbers for deviance analysis. They find that log commits is the single most contributing factor, with second most important being the languages at less than 1%:

```r id=c384c48a-7e1a-4362-bb5e-9fc84d0472f2
anova(nbfit)
```

[__out-0.csv][nextjournal#output#c384c48a-7e1a-4362-bb5e-9fc84d0472f2#__out-0.csv]

Well, this looks different indeed. This is because we have different order of the control varibles and `anova` checks type-I tests so order matters. If we change the order to correspond to the original manuscript, we can replicate:

```r id=ea87e534-201f-4a02-9864-afa527d5e926
nbfit_order_fixed = glm.nb(bcommits~lcommits+lmax_commit_age+ltins+ldevs+language, contrasts = list(language = contr.Weights(Y$language)), data=Y)
anova(nbfit_order_fixed)
```

[__out-0.csv][nextjournal#output#ea87e534-201f-4a02-9864-afa527d5e926#__out-0.csv]

However, proper way would be to use `Anova` function which is order independent as it does type-II tests:

```r id=cdd13d43-6b99-45b8-8c14-d50d1d6d481a
Anova(nbfit)
```

[__out-0.csv][nextjournal#output#cdd13d43-6b99-45b8-8c14-d50d1d6d481a#__out-0.csv]

## Which Language Properties Relate to Defects (chapter 3, RQ 2)

First, let's drop Typescript, which now belongs to the `Other` category since we are not interested in it:

```r id=2a4ef05e-0de6-4beb-b55f-a7dc1a91227d
newSha$combined = newSha$combinedOriginal
newSha$devs = newSha$committer
X = summarizeByLanguage(newSha)
Y = logTransform(X, log10, log)
Y = Y %>% filter(combined != "Other")
Y = droplevels(Y)
```

Let's check the weights first, as we did in RQ1:

```r id=2bdf86cd-0f2b-4fab-bd36-87261d6fe180
contr.Weights(Y$combined)
```

[__out-0.csv][nextjournal#output#2bdf86cd-0f2b-4fab-bd36-87261d6fe180#__out-0.csv]

Fit the model, similarly to RQ1:

```r id=6920dcff-21c0-48c7-934d-46696eb94a3f
nbfit = glm.nb(bcommits~lmax_commit_age+ltins+ldevs+lcommits+combined, contrasts = list(combined = contr.Weights(Y$combined)), data=Y)
nbfit_r = glm.nb(bcommits~lmax_commit_age+ltins+ldevs+lcommits+combined_r, contrasts = list(combined_r = contr.Weights(Y$combined_r)), data=Y)
# combine them into single result table
result = combineModels(nbfit, nbfit_r, Y$combined)
result
```

[__out-0.csv][nextjournal#output#6920dcff-21c0-48c7-934d-46696eb94a3f#__out-0.csv]

And let's juxtapose this against the original FSE paper:

```r id=6df945d4-528f-4437-80f6-21bd47f22da9
juxt = merge(result, baselineFSE_RQ2(), by = 0, all = T, sort = F)
juxt$ok = checkPValues(juxt, "FSE_pv", "pVal")
juxt
```

[__out-0.csv][nextjournal#output#6df945d4-528f-4437-80f6-21bd47f22da9#__out-0.csv]

Using the original classification, we can repeat the RQ2 results properly. The only difference is the large coefficient we see in commits. This is because apparently this time the FSE paper used natural logarithms everywhere, so we try that too:

Now let's look if this changes when we use the updated classification:

```r id=3d0f85f3-894d-46b0-8d0d-b8cea873fd95
newSha$combined = newSha$combinedCorrected
Xcor = summarizeByLanguage(newSha)
Ycor = logTransform(Xcor, log, log)
Ycor = Ycor %>% filter(combined != "Other")
Ycor = droplevels(Ycor)
nbfit = glm.nb(bcommits~lmax_commit_age+ltins+ldevs+lcommits+combined, contrasts = list(combined = contr.Weights(Ycor$combined)), data=Ycor)
nbfit_r = glm.nb(bcommits~lmax_commit_age+ltins+ldevs+lcommits+combined_r, contrasts = list(combined_r = contr.Weights(Ycor$combined_r)), data=Ycor)
# combine them into single result table
resultReclassified = combineModels(nbfit, nbfit_r, Ycor$combined)
juxt = merge(resultReclassified, baselineFSE_RQ2(), by = 0, all = T, sort = F)
juxt$ok = checkPValues(juxt, "FSE_pv", "pVal")
juxt
```

[__out-0.csv][nextjournal#output#3d0f85f3-894d-46b0-8d0d-b8cea873fd95#__out-0.csv]

We can observe that the log commits coefficient now corresponds to the original value due to the log change. Furthermore we see that the `Pro Sta Str Man` and `Scr Dyn Wea Man` categories are no longer significant and we cannot say anything about the  `Fun Dyn Str Man` category since there are no such languages (but these almost exclusively went to `Fun Dyn Wea Man`, which is signifcant under the reclassification).

To summarize, we were able to replicate RQ2 properly. However, if we correct the errors in language classification into categories, the claims are much weaker - notably procedural managed languages and scripting languages are no longer significantly more likely to contain bugs.

Finally, create the table we will use in the paper:

```r id=d76ab2af-9134-4030-92c1-a08fd16109ad
output_RQ2_table(result, resultReclassified)
```

## Does Language Defect Proneness Depend on Domain (chapter 3, RQ3)

The artifact we obtained contained no code to answer this question, so will attempt to figure the code out ourselves.

Let's try to recreate the heatmaps from RQ3 original paper. First prepare the data:

```r id=a8179331-f861-4eb6-996c-8f6c20335fc6
langs = length(unique(Y$language))
domains = length(unique(Y$domain))

ratio = matrix(0, langs, domains)
projects = matrix(0, langs, domains)
commits = matrix(0, langs, domains)
bcommits = matrix(0, langs, domains)

for (i in 1:nrow(Y)) {
    r = Y[i,]
    langIndex = as.numeric(r$language)
    domainIndex = as.numeric(r$domain)
    projects[langIndex, domainIndex] = projects[langIndex, domainIndex] + 1
    commits[langIndex, domainIndex] = commits[langIndex, domainIndex] + r$commits
    bcommits[langIndex, domainIndex] = bcommits[langIndex, domainIndex] + r$bcommits
}

ratio = bcommits / commits
```

Now do the heatmap:

```r id=7ab16acc-7b3f-4be8-8719-3bf561728667
heatmap.2(ratio, dendrogram = "none", cellnote = round(ratio,2), trace = "none", labRow = levels(Y$language), labCol = levels(Y$domain), Rowv = NA, Colv = NA, notecol = "red", col = gray(256:0 / 256 ))
```

![__out-0][nextjournal#output#7ab16acc-7b3f-4be8-8719-3bf561728667#__out-0]

Visual comparison, since this is all we have because no read data on the heatmaps was in the artifact either does confirm that this is very similar to the first heatmap in RQ3 of the original paper. After that the paper argues that outliers needed to be removed, where outliers are projects with bug densities below 10th percentile and above 90th percentile. We therefore remove these and create the second heatmap, first creating the data in the same way as before:

```r id=840221e6-8efb-4bcd-80e4-756d7ee1c704
ratioVect = Y$bcommits / Y$commits
q10 = quantile(ratioVect, 0.1)
q90 = quantile(ratioVect, 0.9)
Y__ = Y[(ratioVect > q10) & (ratioVect < q90),]

ratio_ = matrix(0, langs, domains)
projects_ = matrix(0, langs, domains)
commits_ = matrix(0, langs, domains)
bcommits_ = matrix(0, langs, domains)

for (i in 1:nrow(Y__)) {
    r = Y__[i,]
    rr = r$bcommits / r$commits 
    langIndex = as.numeric(r$language)
    domainIndex = as.numeric(r$domain)
    projects_[langIndex, domainIndex] = projects[langIndex, domainIndex] + 1
    commits_[langIndex, domainIndex] = commits[langIndex, domainIndex] + r$commits
    bcommits_[langIndex, domainIndex] = bcommits[langIndex, domainIndex] + r$bcommits
}

ratio_ = bcommits_ / commits_
```

How much data did we shed?

```r id=f018b3a1-e7a3-4d3a-8c34-8dcf1073f753
cat(paste("Records:          ", round(100 - nrow(Y__) / nrow(Y) * 100, 2), "%\n"))
cat(paste("Buggy commits:    ", round(100 - sum(Y__$bcommits) / sum(Y$bcommits) * 100, 2), "%\n"))
cat(paste("Commits:          ", round(100 - sum(Y__$commits) / sum(Y$commits) * 100, 2), "%\n"))
```

Ok, that is kind of substantial. But that happens when you go 0.1 and 0.9%, should be \~ 20% of the data.

The heatmap:

```r id=e6e68ace-7464-4511-8c9d-57109c202ade
heatmap.2(ratio_, dendrogram = "none", cellnote = round(ratio_,2), trace = "none", labRow = levels(Y__$language), labCol = levels(Y__$domain), Rowv = NA, Colv = NA, notecol = "red", col = gray(256:0 / 256 ))
```

![__out-0][nextjournal#output#e6e68ace-7464-4511-8c9d-57109c202ade#__out-0]

So, this is different to me. In the original paper, C is strongest in Code Analyzer, while we see C being strongest in Database. Also Perl is very strong in Database, but we see no such thing in the paper.

Note: The paper ignores the other category and instead shows the overall results, but this has no effect on the actual domains.

Frustrated by the fact that we cannot replicate the second heatmap, we were thinking if there are other means by which we can support or deny the claims of the original RQ3. This claim is that there is no general relationship between domain and language defect proness. To demonstrate this, we can try the glm from RQ1 and RQ2, but this time see domains. If we do not find significant correlation, then since we have found significant correlation for languages already, it would follow that domains have no general effect.

Let's first summarize the data again:

```r id=3f203c5b-2b45-42c2-ad5d-c75dbc8bde98
newSha$combined = newSha$combinedCorrected
newSha$devs = newSha$committer
X = summarizeByLanguage(newSha)
Y = logTransform(X, log, log)
# discard the other domain
Y = Y %>% filter(domain != "Other")
Y = droplevels(Y)
```

And fit the model, this time showing the domain instead of languages or language categories. However, we do two extra things - we change the contrasts to zero sum contrasts, which makes more sense, and we adjust for multiple hypothesis using the Bonferroni method - see re-analysis.Rmd or the paper for more details on why.

```r id=be6f7148-73c9-40ae-bc72-6514b7cbe4b3
nbfit = glm.nb(bcommits~lmax_commit_age+ltins+ldevs+lcommits+domain, contrasts = list(domain = "contr.sum"), data=Y)
nbfit_r = glm.nb(bcommits~lmax_commit_age+ltins+ldevs+lcommits+domain_r, contrasts = list(domain_r = "contr.sum"), data=Y)
# combine them into single result table
result = combineModels(nbfit, nbfit_r, Y$domain, pValAdjust = "bonferroni")
result
```

[__out-0.csv][nextjournal#output#be6f7148-73c9-40ae-bc72-6514b7cbe4b3#__out-0.csv]

Looking at the results above, none of the domains is significant at level `0.01` so we can say that our tests supports the claim made by the original authors and move to the last RQ. But before doing just that, we should generate a lateX table from our result since this table goes into the paper:

```r id=0597ff23-6b63-4368-a49e-a9d3ef5eab7e
result$pVal = lessThanPvCheck001(round(result$pVal, 2))
for (i in 1:nrow(result)) {
    out(paste0("rqIIIname",as.roman(i)), rownames(result)[[i]])
    out(paste0("rqIIIcoef", as.roman(i)), result$coef[[i]])
    out(paste0("rqIIIpv", as.roman(i)), result$pVal[[i]])
}
```

## What is the relation between language and bug category? (chapter 3, RQ4)

There was no column in the dataset with the same bug type labels in Table 5 in the paper. Our closest approximation to the authors' bug categories classifies the commits with Algo, Concurrency, Memory, and Programming in btype1 with those respective labels; it classifies the commits with Security, Performance, and Failure from btype2; and it classifies commits as Other according to whether they are labeled as such in btype1 and btype2.

```r id=47022958-8e10-454e-9ab6-8bd98f49c108

btype_list <- mapply(c, newSha$btype1, newSha$btype2, SIMPLIFY=F)

nb_fun <- function(l) {
  !("NB" %in% l)
}

btype_justbugs <- Filter(nb_fun, btype_list)
numjustbugs    <- length(btype_justbugs)
check(numjustbugs == 559186)

# Make sure btype1 != btype2
# This will rule out list("Other", "Other")
different_btypes <- function(l) {
  length(unique(l)) == 2
}

# rows with one btype listed as "Other" are
# not overlapping
other_fun <- function(l) {
  !("Other" %in% l)
}

overlap    <- Filter(other_fun, btype_justbugs)
overlap    <- Filter(different_btypes, overlap)
numoverlap <- length(overlap)
check(numoverlap == 7417)

percent_overlap <- round((numoverlap / numjustbugs)*100, digits=2)
```

It is unclear whether the bug categories are meant to be mutually exclusive--but the sum of their percent counts is 104.44%.

Of the 559186 buggy commits in newSha, 1.33% of them have two different bug categories assigned to them, neither of which is "Other".

We attempted to recreate the numbers in Table 5 with the unmodified data in newSha.

```r id=b311cf2a-351f-4645-a50f-49c3d2461c1b

T5_bug_type <- c("Algorithm", "Concurrency", "Memory", "Programming", "Security", "Performance", "Failure", "Unknown")
bugtype_count <- c(606, 11111, 30437, 495013, 11235, 8651, 21079, 5792)
percent_count <- c(0.11, 1.99, 5.44, 88.53, 2.01, 1.55, 3.77, 1.04)
# Calculate highest possible value for each category in our data set

btype_algo <- nrow(newSha[newSha$btype1 == "Algo",])
btype_conc <- nrow(newSha[newSha$btype1 == "Concurrency",])
btype_mem  <- nrow(newSha[newSha$btype1 == "Memory",])
btype_prog <- nrow(newSha[newSha$btype1 == "Programming",])
btype_sec  <- nrow(newSha[newSha$btype2 == "Security",])
btype_perf <- nrow(newSha[newSha$btype2 == "Performance",])
btype_fail <- nrow(newSha[newSha$btype2 == "Failure",]) # There is no "FAILURE" category in bug_type
btype_unknown <- nrow(newSha[newSha$btype1 == "Other" & newSha$btype2 == "Other",])

our_bugtype_count <- c(btype_algo, btype_conc, btype_mem, btype_prog, btype_sec, btype_perf, btype_fail, btype_unknown)

btype_total = sum(newSha$isbug)
our_percent = round((our_bugtype_count / btype_total)*100, digits=2)
factordiff = mapply(function(X, Y) { round((Y / X), digits=4) }, bugtype_count, our_bugtype_count)
countdiff   = mapply(function(X, Y) { Y - X }, bugtype_count, our_bugtype_count)

bugpercenttotal_known <- sum(percent_count[1:7])
check(bugpercenttotal_known == 103.4)
bugpercenttotal <- sum(percent_count)
check(bugpercenttotal == 104.44)

t5_bug_categories <- data.frame(T5_bug_type, btype_count=bugtype_count, percent=percent_count, our_btype_count=our_bugtype_count, our_percent, diff_factor=factordiff, count_difference=countdiff)
t5_bug_categories
```

[__out-0.csv][nextjournal#output#b311cf2a-351f-4645-a50f-49c3d2461c1b#__out-0.csv]

These differences in the composition of the data shared with us versus that reported by the authors make the data incomparable.

```r id=8a8f1376-ee49-4d4f-906f-e14d53e6dd94
remove(WORKING_DIR)
```

## Summary

We are mostly able to verify the basic data collection steps, although we have found discrepancies in some numbers reported as well as two-way errors in the summarization of the unique project-commit-file data in `everything` and `project-commit-language` in `newSha`, but overall the data in the artifact seem to be reasonably complete to attempt to repeat the analysis.

We were not able to repeat the project domain and bug category classifications since the artifact contain no code towards these and the description provided in the paper was not detailed enough for us to repeat from scratch.

The paper had 4 claims, of which:

We were able to completely replicate the first claim (language bug proneness) if we assume the typo in one significance level. We have found minor inconsistencies in the paper which do not have effect on the claim.

We were able to replicate the second claim (language categories), however we found mistakes in the language classification. When we corrected for them the original claim was only partially valid.

We were not able to replicate the third claim (language domains) because the data cleaning described in the paper provided not complete description and our data afterward differed significantly from the original. We have however used another method to the same end.

We were not able to replicate the fourth claim (bug categories). Input data to this problem was not available and no detailed description on how to recreate them was available in the artifact or the paper.

# R Notebook (re-analysis.Rmd)

This is our attempt at re-analysis. We only concentrate on RQ1.

First, because this `Rmd` file can be executed by external script to allow automatic generation of all possible permutations of our re-analysis cleaning steps, we define certain variables that conditionally enable features of this document. By default they are all set to `TRUE`:

```r id=216cfdf3-5285-4e88-b9f7-2c59fdb86d21
if (! exists("REMOVE_DUPLICATES"))
    REMOVE_DUPLICATES = T
if (! exists("REMOVE_TYPESCRIPT"))
    REMOVE_TYPESCRIPT = T
if (! exists("REMOVE_V8"))
    REMOVE_V8 = T
if (! exists("UNCERTAINTY"))
    UNCERTAINTY = T
if (! exists("USE_AUTHORS_INSTEAD_OF_COMMITTERS"))
    USE_AUTHORS_INSTEAD_COMMITTERS = T
# sanity check prints
cat(paste("REMOVE_DUPLICATES:              ", REMOVE_DUPLICATES, "\n"))
cat(paste("REMOVE_TYPESCRIPT:              ", REMOVE_TYPESCRIPT, "\n"))
cat(paste("REMOVE_V8:                      ", REMOVE_V8, "\n"))
cat(paste("UNCERTAINTY:                    ", UNCERTAINTY, "\n"))
cat(paste("USE_AUTHORS_INSTEAD_COMMITTERS: ", USE_AUTHORS_INSTEAD_COMMITTERS, "\n"))
```

Next, we setup working dir. Working dir can also be provided by external script, however if it is not provided, we verify that all features are enabled and set it to the default, i.e. `artifact/re-analysis`, where the paper expects the graphs to be found.

```r id=790589d6-69cd-4e2b-9d95-135a3eead0d5
if (! exists("WORKING_DIR")) {
    if (! REMOVE_DUPLICATES & REMOVE_TYPESCRIPT & REMOVE_V8 & UNCERTAINTY & USE_AUTHORS_INSTEAD_COMMITTERS)
        stop("If non-default flags are submitted to the re-analysis notebook, WORKING_DIR must be set.")
    WORKING_DIR = "./artifact/re-analysis"
}
```

Now initialize the script's environment:

```r id=12cd028f-5fde-4c4f-af61-efc94a84b836
# load the file containing the actual implementation details
knitr::opts_chunk$set(echo = FALSE)
source("implementation.R")
initializeEnvironment()
```

## Data Collection

We reuse the cleaning and optimization passes from the repetition and just load the result of that:

```r id=97fbf0ee-a80d-45cf-a490-eac80bb14cf7
data = read.csv("./artifact/repetition/Data/newSha.csv")
initial_data = data
initial_number_of_commits = length(unique(data$sha))
initial_number_of_rows = nrow(data)
everything =loadEverything()
```

## Data Cleaning

### Removing Duplication

First, we remove any duplicates, i.e. same commits that are present in multiple projects. We remove all occurences of commits that have duplicates since we do not know which one to pick to stay and different picks may bias the later analysis.

```r id=e835d074-edc9-40d2-a974-4e598b2a7810
if (REMOVE_DUPLICATES) {
    sha_proj = data %>% dplyr::select(sha, project) %>% group_by(sha, project) %>% dplyr::summarize(n = n())
    duplicates_sha = sha_proj %>% group_by(sha) %>% dplyr::summarize(n = n()) %>% filter(n > 1)
    # this is the number of commits that are present in multiple commits
    num_duplicates_sha = nrow(duplicates_sha)
    check(num_duplicates_sha == 27450)
    # how many projects are affected? 
    dup_repos = data %>% filter(sha %in% duplicates_sha$sha)
    dup_repos = unique(dup_repos$project)
    num_dup_repos = length(dup_repos)
    check(num_dup_repos == 33)
    # determine how many commits are there in the affected projects in total
    commits_by_dup_repos = data %>% filter(project %in% dup_repos) %>% group_by(sha)
    num_commits_by_dup_repos = nrow(commits_by_dup_repos)
    # since we can't do better, exclude all duplicate commits from the dataset
    data = data %>% filter(! sha %in% duplicates_sha$sha);
    # some reporting for the paper
    out("numberOfProjectsWithDuplicates", num_dup_repos)
    out("numDuplicateCommits", num_duplicates_sha)
    out("percentageDuplicateCommits", round(num_duplicates_sha/initial_number_of_commits * 100,2))
    out("percentageDuplicateRowsLost", round(100 - nrow(data) / initial_number_of_rows * 100, 2))
    hist(duplicates_sha$n, breaks = 100)
}
```

![__out-0][nextjournal#output#e835d074-edc9-40d2-a974-4e598b2a7810#__out-0]

By removing the duplicates we may have created project-language pairs that have fewer than 20 commits. The original study excluded such pairs and we see no reason not to repeat this. Let's find any such rows and delete them

```r id=15f12eda-f44b-42ee-be20-5e76df2209c4
if (REMOVE_DUPLICATES) {
    num_rows_before_cleanup = nrow(data)
    not_keep = data %>% group_by(project, language) %>% dplyr::summarize(n = n()) %>% filter(n < 20)
    keep = data %>% group_by(project, language) %>% dplyr::summarize(n = n()) %>% filter(n >= 20)
    data = keep %>% inner_join(data, by=c("project", "language"))
    out("smallProjectCommits", nrow(not_keep))
    out("percentageDuplicationReadjustmentRowsLost", round(100 - nrow(data)/num_rows_before_cleanup * 100, 2))
}
```

### Removing Typescript

The Typescript language was released in October 1, 2012 and is the smallest language in the study. However, its first commit in the study is dated 2003. What is happening here is that that the `.ts` extension is used for translation files, not Typescript, which was not detected by the original study.

```r id=0992f2ec-40e7-4ac2-b363-49afa188b1b2
if (REMOVE_TYPESCRIPT) {
    ts = data %>% filter(language == "Typescript")
    ts_rows_num = nrow(ts)
    ts_proj_num = length(unique(ts$project))
    check(ts_proj_num == 53)
    out("initialNumTSProjects", ts_proj_num)
    ts_commits_num <- length(unique(ts$sha))  
    check(ts_commits_num == 10105)
    out("initialNumTSCommits", ts_commits_num)
    ts_dates <- sort(unique(ts$commit_date))  # "2003-03-01"
    check(ts_dates[1] == "2003-03-21")
    out("tsFirstCommit", ts_dates[1])
}
```

We now remove from TS what we manually confirmed to be translations and not actual TypeScript:

```r id=3a9e8c3d-46e3-4881-84d1-b15f729157c6
if (REMOVE_TYPESCRIPT) {
  translation_projects <- c("mythtv", "hw", "SpeedCrunch", "qBittorrent", "mumble", "pokerth", "hydrogen", "unetbootin",
                          "tiled", "goldendict", "zf2", "Cockatrice", "tagainijisho", "focuswriter", "LibreCAD", 
                          "razor-qt", "qupzilla", "tomahawk", "ppcoin", "mirall", "MuseScore", "shotcut")
  ts = ts %>% filter(! project %in% translation_projects)
  check(length(unique(ts$sha)) == 4456)
  ts = ts %>% 
      filter(!str_detect(project, "coin")) %>%  # remove all the *coin  projects
      filter(!str_detect(project, "Coin")) %>%  # remove all the *Coin  projects
      filter(!str_detect(project, "change"))  # remove all the *change  projects
  # TODO why are these in remaining??? 
  remaining_translations <- c("antimicro", "identifi", "octopi", "pcbsd", "subsurface", "ttrss", "wps_i18n")
  ts = ts %>% filter(!project %in% remaining_translations)
  real_ts_num_proj <- length(unique(ts$project)) 
  check(real_ts_num_proj == 16)
  out("realTSProjNum", real_ts_num_proj)
  real_ts_num_commit <- length(unique(ts$sha))
  check(real_ts_num_commit == 3782)
  out("realTSCommitsNum", real_ts_num_commit)
  real_ts_proj <- unique(ts$project)
}
```

Another issue with Typescript is that a lot of its files are only type definitions, and contain no actual code in Typescript but headers for Javascript functions and classes elsewhere. We should exclude these as well:

```r id=ca065d26-97b7-4aa2-947b-3017be5ba5fd
if (REMOVE_TYPESCRIPT) {
  # typescript-node-definitions, DefinitelyTyped, tsd
  # DEPRECATED: TSD is deprecated, please use Typings and see this issue for more information.

  tdefs1 = ts %>% filter(project=="typescript-node-definitions") 
  tdefs2 = ts %>% filter(project=="DefinitelyTyped") 
  tdefs3 = ts %>% filter(project=="tsd")
  sts <- length(unique(ts$sha))
  st1 <- length(unique(tdefs1$sha))
  st2 <- length(unique(tdefs2$sha))
  st3 <- length(unique(tdefs3$sha))

  ratio <- round((st1 + st2 + st3)/sts*100, 1)
  check(ratio == 34.6)
  out("ratioOfTypeDefTSCommits", ratio)
  
  ts = ts %>% filter(project !="typescript-node-definitions") %>% filter(project !="DefinitelyTyped") %>% filter(project !="tsd")
}
```

So what are we left with?

```r id=08d05a34-4457-42c5-b8e3-4f4387445335
if (REMOVE_TYPESCRIPT) {
   ts_valid_rows_num = nrow(ts)
   ts_valid_commits = length(unique(ts$sha))
   ts_valid_projects = length(unique(ts$project))
   out("tsValidRows", ts_valid_rows_num)
   out("tsValidCommits", ts_valid_commits)
   out("tsValidProjects", ts_valid_projects)
}
```

We are down to only 13 projects out of 50 and given the fact Typescript was the smallest language to begin with and how much of its commits we had to remove with minimal effort, the only safe thing we can do is to exclude Typescript from the analysis completely:

```r id=557942a2-05a3-440d-92ad-fee3bacb0147
if (REMOVE_TYPESCRIPT) {
    num_rows_before_ts = nrow(data)
    data = data %>% filter(language != "Typescript")
    out("percentageTypescriptRowsLost", round(100 - nrow(data) / num_rows_before_ts * 100, 2))
    data$language = factor(as.character(data$language))
}
```

## Removing V8

The V8 project is plagued with errors and inaccuracies:

```r id=75268cf0-cba3-450f-99b5-4cc82a0d0606
if (REMOVE_V8) {
    v8 = everything %>% filter(project == "v8")
    out("vCCommits", length(unique(v8[v8$tag == "c",]$sha)))
    out("vCppCommits", length(unique(v8[v8$tag == "cpp",]$sha)))
    out("vJavascriptCommits", length(unique(v8[v8$tag == "javascript",]$sha)))
    out("vPythonCommits", length(unique(v8[v8$tag == "python",]$sha)))
}
```

The paper ignores all of V8's C++ files (the `.cc` and `.h` file extensions are ignored) and classifies V8 as a JavaScript project. This is obviously wrong and since V8 is one of the larger projects, may skew the analysis substantially. To avoid this, we remove V8 from the data:

```r id=e602305d-d33e-44c8-8253-53eb54113c29
if (REMOVE_V8) {
    num_rows_before_v8 = nrow(data)
    data = data %>% filter(project != "v8")
    out("percentageVEightRowsLost", round(100 - nrow(data) / num_rows_before_v8 * 100, 2))
}
```

### Summary

Let's summarize the final numbers after all cleaning:

```r id=7a4e89d3-7f36-43b2-8bca-9ea2fc4ccf6b
newShaNum <- length(unique(data$sha))
ratioSha <- round(100 - (newShaNum/initial_number_of_commits*100),2)
out("finalNumSha", newShaNum)
out("finalNumShaMio", round(newShaNum/1000/1000,1))
out("ratioReducedSha", ratioSha)
out("ratioReducedShaRows", round(100 - nrow(data) / nrow(initial_data) * 100, 2))

f_number_of_projects <- length(unique(data$project))
check(f_number_of_projects == 719) 
out("finalNumberOfProjectsIncluded", f_number_of_projects)

f_sloc <- sum(data$insertion) - sum(data$deletion)
f_sloc_mio <- round(f_sloc / 1000000, 1)
check(f_sloc_mio == 58.2)
out("finalSlocMio", f_sloc_mio)

f_number_authors <- length(unique(data$author))
check(f_number_authors == 46204)
out("finalNumberAuthors", round(f_number_authors/1000,0))

f_bugFixes = data %>% filter(isbug == 1)
f_numberOfBugFixes <-  length(unique(f_bugFixes$sha))
out("finalNumberOfBugFixes", f_numberOfBugFixes)
```

Let's see how much data we shed in different categories:

```r id=d40b14e5-6271-447f-8880-028e9305ae69
cat(paste("Commits (unique):     ", round(100 - length(unique(data$sha)) / length(unique(initial_data$sha)) * 100, 2), "%\n"))
cat(paste("Commits (rows):       ", round(100 - nrow(data) / nrow(initial_data) * 100, 2), "%\n"))
cat(paste("Buggy commits (rows): ", round(100 - sum(data$isbug) / sum(initial_data$isbug) * 100, 2), "%\n"))
cat(paste("Projects:             ", round(100 - length(unique(data$project)) / length(unique(initial_data$project)) * 100, 2), "%\n"))
cat(paste("SLOC:                 ", round(100 - sum(data$insertion - data$deletion) / sum(initial_data$insertion - initial_data$deletion) * 100, 2), "%\n"))
cat(paste("Languages:            ", round(100 - length(unique(data$language)) / length(unique(initial_data$language)) * 100, 2), "%\n"))
cat(paste("Authors:              ", round(100 - length(unique(data$author)) / length(unique(initial_data$author)) * 100, 2), "%\n"))
```

## Summarization

Summarize the data for the graphs and modelling as we did in the repetition:

```r id=9fa8a6ac-b8f2-4575-9498-b67e0b306d41
data$combined = data$combinedOriginal # does not matter
if (USE_AUTHORS_INSTEAD_COMMITTERS) {
    data$devs = data$author
} else {
    data$devs = data$committer
}
X = summarizeByLanguage(data)
Y = logTransform(X, log, log)
```

## Graphs

Let's do some graphs:

### Commits vs. Bugfixes.

```r id=490f0c1f-fd45-4847-9a31-3cc789f2b9b8
data %>% group_by(language) %>% summarize(commits = n(), bugfixes = sum(isbug)) -> df

ggplot(data=df, aes(x=commits, y=bugfixes)) + 
  geom_smooth(method='lm',size=.5, formula = y ~ x)  +
  geom_point()  + 
  geom_text_repel(aes(label=language), segment.color = 'grey50', fontface = 'bold', size = 5) + 
  theme_light() +
  theme(text = element_text(size = 15)) +
  labs(y="Bug-fixing commits", x = "Commits") +
  scale_y_log10(breaks = trans_breaks("log10", function(x) 10^x),
                labels = trans_format("log10", math_format(10^.x))) + 
  scale_x_log10(breaks = trans_breaks("log10", function(x) 10^x),
                  labels = trans_format("log10", math_format(10^.x))) -> p

ggsave(paste(WORKING_DIR, "/Figures/commits.pdf", sep = ""))
print(p)
```

![__out-0][nextjournal#output#490f0c1f-fd45-4847-9a31-3cc789f2b9b8#__out-0]

### Bug rate vs project age for selected languages

```r id=0034f7d6-85c5-4f6d-b099-4ae229f88814
X %>% 
  dplyr::select(project, language, commits, bcommits,max_commit_age) %>% 
  mutate(br=round(bcommits/commits*100,1)) -> df

most_commits <- .9 * sum(X$commits)
df %>% arrange(desc(commits)) -> df

# find the number of project that would comprise over 90% of the entire dataset
accumulatedCommits = 0
i = 0
while (accumulatedCommits <= most_commits) {
    i = i + 1
    accumulatedCommits = accumulatedCommits + df[i, 3]
}
# verify that we have the correct nunber
check(sum(df[1:i,3]) > most_commits) 
check(sum(df[1:i-1,3]) <= most_commits) 

df[1:i,] ->most_df

most_df %>% filter(language %in% c("C", "C++","Clojure","Haskell","Objective-C","Scala")) -> most_df2

br_mean <- most_df2 %>% group_by(language) %>% summarise(br=mean(br))
age_mean <- most_df2 %>% group_by(language) %>% summarise(age=mean(max_commit_age))

ggplot(data=most_df2, aes(x=max_commit_age,y=br,color=language)) + geom_point() +
  geom_hline(aes(yintercept=br, color='black'), size=.25, br_mean) +
  geom_vline(aes(xintercept=age), size=.25, age_mean) +
   theme_light() +
  theme(text = element_text(size=20)) +
  scale_x_log10(breaks = trans_breaks("log10", function(x) 10^x), 
                  labels = trans_format("log10", math_format(10^.x))) +
                labs(x="Project age (days)", y="Bug rate (percent of commits)") +
                theme(legend.position = "none") +
                facet_wrap(~language, ncol=2) -> p

X %>% 
  mutate(br=round(bcommits/commits*100,1)) %>% 
  group_by(language) %>% 
  summarize(mbr = mean(br), mage = mean(max_commit_age)) %>% 
  arrange(desc(mbr)) -> mbr

ggsave(paste(WORKING_DIR, "/Figures/age.pdf", sep = ""), width=8, height=8)
print(p)
```

![__out-0][nextjournal#output#0034f7d6-85c5-4f6d-b099-4ae229f88814#__out-0]

### Developer behavior

```r id=497f38e6-82bf-44d4-90c5-4e25ba2fb8bd
data %>% group_by(author) %>% summarize(n=n())  %>% arrange(desc(n)) -> devs

most_commits_bydev <- .9*sum(devs$n)

# find the number of developers accounting for <= 90% of the entire dataset
accumulatedCommits = 0
d = 0
while (accumulatedCommits <= most_commits_bydev) {
    d = d + 1
    accumulatedCommits = accumulatedCommits + devs[d, 2]
}

# verify that we have the correct nunber
check(sum(devs[1:d-1,2]) < most_commits_bydev) 
check(sum(devs[1:d,2]) > most_commits_bydev)

devs_prolific <- devs$author[1:d]

# Show the number of languages the most prolific authors committed for
data %>% filter(author %in% devs_prolific) -> devs_prolific
devs_prolific = devs_prolific %>% group_by(author,language) %>% summarize(n=n()) %>% summarize(l=n())  %>% arrange(desc(l))
devs_prolific
hist(devs_prolific$l)
```

![__out-1][nextjournal#output#497f38e6-82bf-44d4-90c5-4e25ba2fb8bd#__out-1]

[__out-0.csv][nextjournal#output#497f38e6-82bf-44d4-90c5-4e25ba2fb8bd#__out-0.csv]

## Modelling

### Basic analyses & graphs of the dataset

```r id=a69c52bc-3752-4fb0-8996-682e07f4afab
dimnames(X)[[2]]
pairs(X[,3:7], gap=0, pch='.')
```

![__out-0][nextjournal#output#a69c52bc-3752-4fb0-8996-682e07f4afab#__out-0]

Let's log transform the data:

```r id=a65ca8e7-abca-4c4d-bef4-355324507c99
Y = logTransform(X)
pairs(Y[,2:6], gap=0, pch='.')
```

![__out-0][nextjournal#output#a65ca8e7-abca-4c4d-bef4-355324507c99#__out-0]

### Language Frequencies

Explore the relative frequencies of the languages--how many projects use them in the dataset

```r id=d24d3172-b9a3-49f0-80a3-04793815e689
sort(table(Y$language))
```

The dataset contains 17 languages (16, if we remove TypeScript). They were represented by unevenly distributed numbers of projects (from 23 for Perl to 199 for Javascript).

Explore the relationship between the languages, and other measurements

```r id=73aceb78-24b4-4c9d-bad7-01cf035d3d99
# (devs, tins, max_commit_age, commits, bcommits)
boxplot(split(Y$lbcommits, Y$language), las=2, main="Log(bugged commits, 0 replaced with 0.5)")
par(mfrow=c(1,2), oma=c(1,1,1,1), mar=c(5,1,2,1))
boxplot(split(Y$ldevs, Y$language), las=2, main="Log(devs)")
boxplot(split(Y$ltins, Y$language), las=2, main="Log(tins)")
boxplot(split(Y$lmax_commit_age, Y$language), las=2,
        main="Log(max_commit_age)")
boxplot(split(Y$lcommits, Y$language), las=2, main="Log(commits)")
```

![__out-2][nextjournal#output#73aceb78-24b4-4c9d-bad7-01cf035d3d99#__out-2]

![__out-0][nextjournal#output#73aceb78-24b4-4c9d-bad7-01cf035d3d99#__out-0]

![__out-1][nextjournal#output#73aceb78-24b4-4c9d-bad7-01cf035d3d99#__out-1]

The distribution of bug-commits varied between the languages, but so did the distributions of the other measurements. It is difficult to see direct associations between languages and bugs from bivariate plots.

### Fitting the Negative Binomial

Let's now fit the negative binomial and compare to the original paper - this is the same code as in replication:

```r id=46e51b59-b929-4539-b343-81af11447fe5
nbfit = glm.nb(bcommits~lmax_commit_age+ltins+ldevs+lcommits+language, contrasts = list(language = contr.Weights(Y$language)), data=Y)
nbfit_r = glm.nb(bcommits~lmax_commit_age+ltins+ldevs+lcommits+language_r, contrasts = list(language_r = contr.Weights(Y$language_r)), data=Y)
# combine them into single result table
resultWeighted = combineModels(nbfit, nbfit_r, Y$language)
juxtWeighted = merge(resultWeighted, baselineFSE_RQ1(), by = 0, all = T, sort = F)
juxtWeighted$ok = checkPValues(juxtWeighted, "FSE_pv", "pVal")
juxtWeighted
```

[__out-0.csv][nextjournal#output#46e51b59-b929-4539-b343-81af11447fe5#__out-0.csv]

And we see that just cleaning the data did invalidate several of the language claims made by the original paper.

### Fitting the zero-Sum Contrasts Negative Binomial

The zero-sum contrasts are preferred to weighted contrasts for their better stability. Let's look at how they look:

```r id=f35490a1-51cd-4200-96ab-1dc14d265847
contr.sum(length(levels(Y$language)))
```

[__out-0.csv][nextjournal#output#f35490a1-51cd-4200-96ab-1dc14d265847#__out-0.csv]

And let's fit the model:

```r id=24514db5-6b38-4647-ae6e-6af8dd632f6a
nbfit = glm.nb(bcommits~lmax_commit_age+ltins+ldevs+lcommits+language, contrasts = list(language = contr.sum), data=Y)
nbfit_r = glm.nb(bcommits~lmax_commit_age+ltins+ldevs+lcommits+language_r, contrasts = list(language_r = contr.sum), data=Y)
# combine them into single result table
resultZeroSum = combineModels(nbfit, nbfit_r, Y$language)
juxtZeroSum = merge(resultZeroSum, baselineFSE_RQ1(), by = 0, all = T, sort = F)
juxtZeroSum$ok = checkPValues(juxtZeroSum, "FSE_pv", "pVal")
juxtZeroSum
```

[__out-0.csv][nextjournal#output#24514db5-6b38-4647-ae6e-6af8dd632f6a#__out-0.csv]

Some invalidatuions still.

### Fit Negative Binomial regression without languages and compare the full and the reduced models

```r id=9bfc7799-00ce-4fdb-bff3-ac820c06add1
nbfit_reduced <- glm.nb(bcommits~lmax_commit_age+ltins+ldevs+lcommits, data=Y)
summary(nbfit_reduced)
```

Comparing two nested models, i.e.$H_0$: reduced model vs $H_a$: full model, using F-test:

```r id=fe2273fd-38ef-443a-b042-e3ad1045e151
anova(nbfit_reduced, nbfit)
```

[__out-0.csv][nextjournal#output#fe2273fd-38ef-443a-b042-e3ad1045e151#__out-0.csv]

```r id=c6dced07-3fab-4a37-a973-3aa1e75873dd
cat("AIC, full:", AIC(nbfit), "\n")
cat("AIC, reduced:", AIC(nbfit_reduced), "\n")
cat("BIC, full:", BIC(nbfit), "\n")
cat("BIC, reduced:", BIC(nbfit_reduced), "\n")
```

The difference between the models is borderline.

### Adjusting for Multiple Hypothesis

The original paper just compares the p-Values against thresholds, which is not correct way to do. In the presence of multiple hypothesis, the p-Values must be adjusted. There are various ways to adjust the p-Values, namely the Bonferroni and FDR (Benjamini & Hochberg). The FDR adjustment is more permissive and the Bonferrioni is the more conservative one. What we can do now is to revisit the juxtaposed tables and add extra columns for the FDR and Bonferroni adjustments:

```r id=0b4d60f3-b305-40ee-ac43-9f90d12a057b
# the the pValues for the predictions, not for the control variables since these are ignored by the adjustments
pValWeighted = juxtWeighted$pVal[6:(5 + length(unique(Y$language)))]
# create the empty vectors with NAs instead of pValues
fdrWeighted = rep(NA, nrow(juxtWeighted))
bonfWeighted = rep(NA, nrow(juxtWeighted))
# update the relevant parts of the vectors with the adjusted pValues
fdrWeighted[6:(5+ length(pValWeighted))] = round(p.adjust(pValWeighted, "fdr"), 3)
bonfWeighted[6:(5+ length(pValWeighted))] = round(p.adjust(pValWeighted, "bonferroni"), 3)
# add the columns to the juxtaposed tables
juxtWeighted$pVal_fdr = fdrWeighted
juxtWeighted$pVal_bonf = bonfWeighted
# now remove the old ok column and add the ok, ok_fdr and ok_bonf columns
juxtWeighted = juxtWeighted %>% dplyr::select(-(ok))
# add the columns
juxtWeighted$ok = checkPValues(juxtWeighted, "FSE_pv", "pVal")
juxtWeighted$ok_fdr = checkPValuesLevel(juxtWeighted, "FSE_pv", "pVal_fdr", 0.01)
juxtWeighted$ok_bonf = checkPValuesLevel(juxtWeighted, "FSE_pv", "pVal_bonf", 0.01)
juxtWeighted
```

[__out-0.csv][nextjournal#output#0b4d60f3-b305-40ee-ac43-9f90d12a057b#__out-0.csv]

With the exception of the last language (Coffeescript), the FDR and Bonferroni are the same and they actually invalidate some of the original predictions. Let's look at the zero-sum contrasts version now:

```r id=430c34d0-686b-4d21-a864-83e02ebb246f
# the the pValues for the predictions, not for the control variables since these are ignored by the adjustments
pValZeroSum = juxtZeroSum$pVal[6:(5 + length(unique(Y$language)))]
# create the empty vectors with NAs instead of pValues
fdrZeroSum = rep(NA, nrow(juxtZeroSum))
bonfZeroSum = rep(NA, nrow(juxtZeroSum))
# update the relevant parts of the vectors with the adjusted pValues
fdrZeroSum[6:(5+ length(pValZeroSum))] = round(p.adjust(pValZeroSum, "fdr"), 3)
bonfZeroSum[6:(5+ length(pValZeroSum))] = round(p.adjust(pValZeroSum, "bonferroni"), 3)
# add the columns to the juxtaposed tables
juxtZeroSum$pVal_fdr = fdrZeroSum
juxtZeroSum$pVal_bonf = bonfZeroSum
# now remove the old ok column and add the ok, ok_fdr and ok_bonf columns
juxtZeroSum = juxtZeroSum %>% dplyr::select(-(ok))
# add the columns
juxtZeroSum$ok = checkPValues(juxtZeroSum, "FSE_pv", "pVal")
juxtZeroSum$ok_fdr = checkPValuesLevel(juxtZeroSum, "FSE_pv", "pVal_fdr", 0.01)
juxtZeroSum$ok_bonf = checkPValuesLevel(juxtZeroSum, "FSE_pv", "pVal_bonf", 0.01)
juxtZeroSum
```

[__out-0.csv][nextjournal#output#430c34d0-686b-4d21-a864-83e02ebb246f#__out-0.csv]

The situation is fairly similar here.

### Statistical significance vs practical significance, for languages

Since the number of observations is large, statistical significance can be driven by the sample size without being meaningful in practice.

Here we contrast model-based confidence intervals and prediction intervals, on the log and on the original scale

```r id=15e63d41-ea4a-4ab3-bca4-1e3d770ece91
numLanguages = length(unique(Y$language))

# Create a new data structure for prediction
newY <- NULL
for (i in 1:numLanguages) {
  newY <- rbind(newY, 
                data.frame(language=rep(levels(Y$language)[i], 100),
                ldevs=rep(median(Y$ldevs), 100), 
                lcommits=seq(from=min(Y$lcommits), to=max(Y$lcommits), length=100),
                ltins=rep(median(Y$ltins), 100),
                lmax_commit_age=rep(median(Y$lmax_commit_age), 100)))
}
newY$commits <- exp(newY$lcommits)

# Make predictions
pr_nbfit <- predict(nbfit, type="response", newdata=newY, se.fit=TRUE)
newY$pr_mean <- pr_nbfit$fit
newY$pr_se <- pr_nbfit$se.fit
```

Consider languages with the most predicted bugs (C++) and fewest predicted bugs (Clojure). Compute the log CI for C++ and Clojure and the log Prediction CI. Then translate the intervals on the original scale.

```r id=1da1c5f5-1d5a-453e-816b-27650a4f5477

axfont  = 32 # size of axis title font
ticfont = 26 # size of axes' font
legfont = 28 # size of legend title
legtext = 26 # size of legend text
ptitle  = 30 # size of plot title letters

getConfInterval<-function(df,lang) {
  df %>% 
    filter(language==lang)  %>%
    mutate(language = lang,
           x = lcommits, y = log(pr_mean), 
           yhigh = log(pr_mean + qnorm(1-0.01/numLanguages) * pr_se),
           ylow =  log(pr_mean - qnorm(1-0.01/numLanguages) * pr_se)) %>%
    dplyr::select(language, x, y, ylow, yhigh)
}
dfCI <- rbind(getConfInterval(newY, "C++"), getConfInterval(newY, "Clojure"))

getPredInterval<-function(df,lang) {
  df %>% 
    filter(language==lang) %>% 
    mutate(language = lang,
           x = lcommits, y = log(pr_mean), 
           yhigh = log(qnbinom(1-0.01/numLanguages, mu= pr_mean, size= nbfit$theta) ),
           ylow = log(qnbinom(0.01/numLanguages, mu=pr_mean, size=nbfit$theta))) %>%
    dplyr::select(language, x, y, ylow, yhigh)
}
dfPI <- rbind(getPredInterval(newY, "C++"), getPredInterval(newY, "Clojure"))

plotIt <- function(df) {
  ggplot(data = df, aes(x=x,y=y,color=language)) + geom_line() +
    geom_ribbon(aes(ymin=df$ylow, ymax=df$yhigh), linetype=2, alpha=0.1) +
    labs(x="log of commits", y="log of bug-fixing commits") + 
    theme_light() +
    theme(axis.title = element_text(size=axfont),
          axis.text = element_text(size=ticfont),
          plot.title = element_text(hjust = 0.5, size = ptitle))
}

plotIt(dfCI) + theme(legend.position = "none") + ggtitle("(a)") -> p1
plotIt(dfPI) + theme(legend.title = element_text(size=legfont),
                      legend.text = element_text(size=legtext)) +
                      guides(colour = guide_legend(nrow = 1)) + 
                      ggtitle("(b)") -> p2

# Grab the legend to display it separately
tmp    <- ggplot_gtable(ggplot_build(p2))
l      <- which(sapply(tmp$grobs, function(x) x$name) == "guide-box")
legend <- tmp$grobs[[l]]
legend

p2 + theme(legend.position = "none") -> p2

getConfInterval <- function(df,lang) {
  df %>% 
    filter(language==lang)  %>%
    mutate(language = lang,
           x = commits,
           y = pr_mean, 
           yhigh = pr_mean + qnorm(1-0.01/numLanguages) * pr_se,
           ylow =  pr_mean - qnorm(1-0.01/numLanguages)* pr_se ) %>%
    dplyr::select(language, x, y, ylow, yhigh)
}
dfCI <- rbind(getConfInterval(newY, "C++"), getConfInterval(newY, "Clojure"))

getPredInterval<-function(df,lang) {
  df %>% 
    filter(language==lang) %>% 
    mutate(language = lang,
           x = commits, y = pr_mean, 
           yhigh = qnbinom(1-0.01/numLanguages, mu= pr_mean, size= nbfit$theta) ,
           ylow =qnbinom(0.01/numLanguages, mu=pr_mean, size=nbfit$theta)) %>%
    dplyr::select(language, x,y,ylow,yhigh)
}
dfPI <- rbind(getPredInterval(newY, "C++"), getPredInterval(newY, "Clojure"))

plotIt <- function(df) {
  ggplot(data = df, aes(x=x,y=y,color=language)) + geom_line() +
    geom_ribbon(aes(ymin=df$ylow, ymax=df$yhigh), linetype=2, alpha=0.1) +
    theme_light() +
    theme(axis.title = element_text(size=axfont),
          axis.text = element_text(size=ticfont),
          plot.title = element_text(hjust = 0.5, size = ptitle),
          legend.position = "none") +
    labs(x="commits", y="bug-fixing commits")
}
plotIt(dfCI) + xlim(0,800) + ylim(0,400) + ggtitle("(c)") -> p3
plotIt(dfPI) + xlim(0,800) + ylim(0,620) + ggtitle("(d)") -> p4
```

```r id=e9173cce-1fcf-4ea6-a12b-7572370ea761
pdf(paste(WORKING_DIR, "/Figures/intervals.pdf", sep = ""), width=32, height=8)
figs <- grid.arrange(arrangeGrob(p1, p2, p3, p4, nrow=1), arrangeGrob(legend), nrow=2, heights=c(4,1))
dev.off()
```

```r id=b09318ab-6250-4c72-9f55-42429737228b
# Plot predicted means for all the languages
par(mfrow=c(1,2))
# Log scale
plot(log(pr_mean)~lcommits,
     main="log(Exp. values), all languages",
     data=newY[newY$language==levels(Y$language)[1],],
     type="l", 
     ylab="log(bugged commits)")
for (i in 2:numLanguages) {
  lines(log(pr_mean)~lcommits,
     data=newY[newY$language==levels(Y$language)[i],])
}

# Original scale
plot(pr_mean~commits,
     main="Exp. values, all languages",
     data=newY[newY$language==levels(Y$language)[1],],
     type="l", 
     xlim=c(0, 800),
     ylim=c(0,400),
     ylab="bugged commits")
for (i in 2:numLanguages) {
  lines(pr_mean~commits,
     data=newY[newY$language==levels(Y$language)[i],])
}
```

![__out-0][nextjournal#output#b09318ab-6250-4c72-9f55-42429737228b#__out-0]

### Add uncertainty in labels

```r id=842e6de5-df0f-4fec-86f8-7c0b785f9ce9
if (UNCERTAINTY) {
    fp <- 0.36
    fn <- 0.11
    # Function to get parameter values
    getParams <- function(Ystar) {
        # Fit NB with standard contrasts
        nbfit <- glm.nb(bcommits~lmax_commit_age+ltins+ldevs+lcommits+language, 
                           contrasts = list(language = "contr.sum"), data=Ystar)
        s <- summary(nbfit)$coefficients
      
        # Fit the releveled model with standard contrasts, 
        # to get the parameter for Scala
        nbfit_r <- glm.nb(bcommits~lmax_commit_age+ltins+ldevs+lcommits+language_r, 
                         contrasts = list(language_r = "contr.sum"), data=Ystar)
        s_r <- summary(nbfit_r)$coefficients
    
        # Return params, incluing Scala
        out <- c(s[,1], s_r[6, 1])
        names(out) <- c(dimnames(s)[[1]][1:5], levels(Ystar$language))
        out
    }
    numBootstrapIterations = 10000
    # Perform sampling
    set.seed(1)
    paramNum <- numLanguages + 5
    paramsLang <- matrix(rep(NA,paramNum*numBootstrapIterations), nrow=paramNum)
    for (i in 1:numBootstrapIterations) {
        # Sample rows
        Ystar <- Y[sample(1:nrow(Y), replace = T),]
      
        # Adjust for tp and fp:
        # Reduce bcommits by prob fp
        # Increase non-bugged commits by prob fn
        tmp <- rbinom(n=nrow(Ystar), size=Ystar$bcommits, prob=1-fp) +
          rbinom(n=nrow(Ystar), size=(Ystar$commits-Ystar$bcommits), prob=fn)
        Ystar$bcommits <- tmp
      
        # Get parameters
        paramsLang[,i] <- getParams(Ystar)
    }
}
```

To determine whether the bootstrapped values are statistically signifficant, we analyze whether the x and 1-x quantiles both have the same sign. If they do, the result is signifficant in that regard, if the do not, then the results is not statistically signifficant. Similarly to p-values, we can compare using different quantiles. The proper, conservative quantile is 0.01 divided by number of languages tested, the less conservative option is just 0.01.

```r id=5c960584-dddb-4c51-bee0-00fd7721297b
if (UNCERTAINTY) {
    paramsRowNames = getModelRowNames(nbfit, Y$language)
    result = data.frame(
        row.names = paramsRowNames,
        coef = rep(NA, length(paramsRowNames)),
        se = rep(NA, length(paramsRowNames)),
        sig = rep(NA, length(paramsRowNames)),
        sigCons = rep(NA, length(paramsRowNames))
    )
    
    
    par(mfrow=c(2,4))
    quant = 0.01
    quantCons = 0.01 / numLanguages
    for ( i in 1:length(paramsRowNames)) {
        result$coef[[i]] = round(mean(paramsLang[i,]), digits = 2)
        result$se[[i]] = round(sd(paramsLang[i,]), digits = 2)
        qsigns = sign(quantile(paramsLang[i,], probs = c(quant, 1-quant, quantCons, 1-quantCons), na.rm = T))
        result$sig[[i]] = qsigns[[1]] == qsigns[[2]]
        result$sigCons[[i]] = qsigns[[3]] == qsigns[[4]]
        hist(paramsLang[i,],
             xlab=paramsRowNames[i], 
             main=paste("mean =", round(mean(paramsLang[i,]), digits=2))
        )
        abline(v=0, col="red", lwd=2)
    }
}
```

```r id=52c0b24f-3cd6-4ca4-ba41-2f000563abab
if (UNCERTAINTY) {
    result
}
```

[__out-0.csv][nextjournal#output#52c0b24f-3cd6-4ca4-ba41-2f000563abab#__out-0.csv]

Finally, let's juxtapose this to the baseline as usual:

```r id=4403efe0-1c1d-4faf-935b-c111c7644a3f
if (UNCERTAINTY) {
    juxtBootstrap = merge(result, baselineFSE_RQ1(), by = 0, all = T, sort = F)
    juxtBootstrap$ok = checkSignificance(juxtBootstrap, "FSE_pv", "sigCons")
    juxtBootstrap
}
```

[__out-0.csv][nextjournal#output#4403efe0-1c1d-4faf-935b-c111c7644a3f#__out-0.csv]

## More graphs: Bug rate over time

```r id=50218144-c124-476a-9c81-c78f4eed67fd
filter_proj <- function(df,proj,lang) {
   data %>% 
    ungroup() %>%
    filter(project == proj, language == lang) %>% 
    dplyr::select(commit_age, isbug) %>%
    arrange(commit_age) -> bt
  bt %>% 
    group_by(commit_age,isbug) %>% 
    summarize(n = n()) %>% 
    spread(key = "isbug",value = "n") -> bt2
  bt2 %>% 
    mutate(br = round(`0`/(`0`+`1`),2), 
           month = as.integer(commit_age/30)) -> bt3
  bt3  %>% 
    na.omit() %>% 
    group_by(month) %>% 
    summarize(n = n(), brs = sum(br), brm = brs/n) %>%
    mutate(name= paste0(proj," ",lang)) -> prj
  rbind(df, prj)
}

filter_proj(NULL, "linux","C") %>%
filter_proj("mono","C#") %>%
filter_proj("homebrew","Ruby") %>%
filter_proj("WordPress","Php") %>%
filter_proj("salt","Python") %>%
filter_proj("mythtv","C++") %>%
filter_proj("jenkins","Java") %>%
filter_proj("akka","Scala") %>%
filter_proj("rabbitmq-server","Erlang") %>%
filter_proj("brackets","Javascript") %>%
filter_proj("yi","Haskell") %>%
filter_proj("overtone","Clojure")  ->prj

prj %>% ggplot( aes(x = month, y = brm)) +
        geom_point(size=.4) +
        theme_light() +
        geom_smooth(method='lm',size=.5, formula = y ~ x) +
        facet_wrap( ~ name,ncol=3) + labs(x="",y="") +
        xlab("Project lifetime (months)") + ylab("Percent bug-labeled commits") +
        theme(strip.text.x = element_text(size = 14), text = element_text(size=13)) +
        theme(text = element_text(size=20))

        
ggsave(paste(WORKING_DIR, "/Figures/bugspermonth.pdf", sep = ""), width = 8, height = 8, dpi = 100)
```

![__out-0.svg][nextjournal#output#50218144-c124-476a-9c81-c78f4eed67fd#__out-0.svg]

## Commits for top-5 projects by language

```r id=a6bce8cd-c677-41e6-a4a5-b08666417620
data %>% group_by(language, project) %>% summarize(n = n()) %>% arrange(desc(n)) %>% arrange(desc(language)) %>% top_n(5, wt = n) -> projsize_ordered2
projsize_ordered2

projsize_ordered2 %>% ggplot(aes(x = reorder(factor(project), -n), y = n)) + 
  geom_bar(stat="identity") + 
  facet_grid(. ~ language, scales = "free") +
  theme(axis.text.x = element_text(angle = 90, hjust = 1)) +
  labs(x = "Project (by language)", y = "Number of Commits") +
  scale_y_continuous(labels = scales::comma)
```

[__out-0.csv][nextjournal#output#a6bce8cd-c677-41e6-a4a5-b08666417620#__out-0.csv]

![__out-1.svg][nextjournal#output#a6bce8cd-c677-41e6-a4a5-b08666417620#__out-1.svg]

## Authors

```r id=1e7dc2c5-fe49-49d5-a68e-809e405860bd
everything %>% group_by(author, sha) %>% summarize(n=n()) %>% group_by(author) %>% summarise(commits = n()) -> auth
auth %>% arrange(desc(commits))
tot <- sum(auth$commits)
check( tot == 1485049)
top <- sort(auth$commits, dec=T)[1:453]
sum(top)/tot ## 46 %

everything %>% group_by(author, project) %>% summarize(n=n()) %>% 
  group_by(author) %>% summarise(proj = n())   %>% arrange(desc(proj)) -> authproj

summary(authproj)  ## mean 1.2
```

[__out-0.csv][nextjournal#output#1e7dc2c5-fe49-49d5-a68e-809e405860bd#__out-0.csv]

## Writing the results

Finally, let's write the results of our analyses into CSV files so that they can be picked and analyzed later:

```r id=19fbed36-4418-4eaf-aa6a-95df78707aff
weighted = data.frame(
    coef = juxtWeighted$coef,
    se = juxtWeighted$se,
    pVal = round(juxtWeighted$pVal,3),
    pVal_fdr = juxtWeighted$pVal_fdr,
    pVal_bonf = juxtWeighted$pVal_bonf,
    row.names = juxtWeighted$Row.names
)
write.csv(weighted, paste0(WORKING_DIR, "/Data/languages_weighed.csv"))
zeroSum = data.frame(
    coef = juxtZeroSum$coef,
    se = juxtZeroSum$se,
    pVal = round(juxtZeroSum$pVal,3),
    pVal_fdr = juxtZeroSum$pVal_fdr,
    pVal_bonf = juxtZeroSum$pVal_bonf,
    row.names = juxtZeroSum$Row.names
)
write.csv(zeroSum, paste0(WORKING_DIR, "/Data/languages_zeroSum.csv"))
# only store bootstrap if we have actually created it
if (UNCERTAINTY) {
    bootstrap = data.frame(
        coef = juxtBootstrap$coef,
        se = juxtBootstrap$se,
        sig = juxtBootstrap$sig,
        sigCons = juxtBootstrap$sigCons,
        row.names = juxtBootstrap$Row.names
    )
    write.csv(bootstrap, paste0(WORKING_DIR, "/Data/languages_bootstrap.csv"))
}
```

```r id=e44c0fc6-aab8-4709-b803-a0f1a962d96e
remove(WORKING_DIR)
```

# Analysis of the permutations of re-anaysis flags  (permutations.Rmd)

```bash id=ea770f4f-b390-4d54-9422-8c3279b5690f
Rscript --vanilla generate_permutations.R
```

When the permutaions have been geneated, we must analyze them. Clear the flags and set the working directory to permutations root:

```r id=7267c2aa-8f03-4789-83a2-912b91fe3bbd
source("implementation.R")

flags = c(
  REMOVE_DUPLICATES = 0, 
  REMOVE_TYPESCRIPT = 0,
  REMOVE_V8 = 0,
  USE_AUTHORS_INSTEAD_COMMITTERS = 0,
  UNCERTAINTY = 0
)

WORKING_DIR = "./artifact/permutations"
```

First, get the permutation results calculated previously:

* `est` = value of the estimate
* `se` = standard error
* `pv` = `NA` if the result was not signifficant, p-value limit otherwise)

For bootstrap, we get `est`, `se` and `sig` and `sigCons` which are boolean vectors telling us whether the results were significant.

```r id=c9dfb915-c7e5-45f3-b066-7b651aff084f
loadPermutation = function(b, name, permutation) {
    lWeighted = read.csv(paste0(permutation, "/Data/languages_weighed.csv"))
    rownames(lWeighted) = lWeighted$X
    lWeighted = lWeighted %>% dplyr::select(-X)
    lZeroSum = read.csv(paste0(permutation, "/Data/languages_zeroSum.csv"))
    rownames(lZeroSum) = lZeroSum$X
    lZeroSum = lZeroSum %>% dplyr::select(-X)
    bootstrapFilename = paste0(permutation, "/Data/languages_bootstrap.csv")
    if (file.exists(bootstrapFilename)) {
        lBootstrap = read.csv(bootstrapFilename)
        rownames(lBootstrap) = lBootstrap$X
        lBootstrap = lBootstrap %>% dplyr::select(-X)
    } else {
        lBootstrap = NULL
    }
    # and create the result list
    list(
        name = name,
        weighted = lWeighted,
        zeroSum = lZeroSum,
        bootstrap = lBootstrap
    )
}
```

Let's now look at all permutations and loat them:

```r id=7408b823-3242-49f9-be62-901948ba1695
loadAllPermutations = function(baseDir, b = baseline) {
    dirs = list.dirs(baseDir, recursive = F)
    result = list()
    for (d in dirs) {
        pName = strsplit(d, "/")
        pName = pName[[1]][[length(pName[[1]])]]
        if (nchar(pName) != length(flags))
            next()
        tryCatch({
            result[[pName]] = loadPermutation(b, pName, d)
        }, error = function(e) {
            print(e)
            cat("Permutation ", pName, "FAIL\n")
        })
    }
    cat("Total valid permutations: ", length(result))
    result
}
permutations = loadAllPermutations(WORKING_DIR)
```

Let's now create the skeleton for teh RQ1 table in the paper:

```r id=8b81d611-da90-4062-9a3c-8e29e3a300ee
result = baselineFSE_RQ1()
result = mergeDataFrames(result, baselineCACM_RQ1())
result = result %>% dplyr::select(FSE_coef, FSE_pv, CACM_coef, CACM_pv)

repetition = permutations[["00000"]]$weighted %>% dplyr::select(repetition_coef = coef, repetition_pv = pVal)
result = mergeDataFrames(result, repetition)

cleaned = permutations[["11111"]]$weighted %>% dplyr::select(clean_coef = coef, clean_pv = pVal, adjusted_fdr = pVal_fdr, adjusted_bonf = pVal_bonf)
result = mergeDataFrames(result, cleaned)

zeroSum = permutations[["11111"]]$zeroSum %>% dplyr::select(zeroSum_coef = coef, zeroSum_pv = pVal_bonf)
result = mergeDataFrames(result, zeroSum)

bootstrap = permutations[["11111"]]$bootstrap %>% dplyr::select(bootstrap_coef = coef, bootstrap_sig = sigCons)
result = mergeDataFrames(result, bootstrap)

result
```

[__out-0.csv][nextjournal#output#8b81d611-da90-4062-9a3c-8e29e3a300ee#__out-0.csv]

And output it to the rq1 table:

```r id=f5f61678-e60d-49ab-9269-3dbf0761b667
output_RQ1_table_repetition(result)
output_RQ1_table_reanalysis(result)
```

```r id=b23d7d74-9998-45c0-b131-d5109e4178da
remove(WORKING_DIR)
```

# Missing Commits (missing_commits.Rmd)

This separate notebook deals with our analysis of commits not accounted for in the artifact obtained from the original authors.

```r id=8960a280-4fbc-4803-80af-378d5c60a7fd
# load the file containing the actual implementation details
knitr::opts_chunk$set(echo = FALSE)
source("implementation.R")
initializeEnvironment("./artifact/missing-commits")
```

Load the data processed by the repetition and summarize them.

```r id=85463605-dd28-446d-b895-a86cedb1b27f
data = read.csv("./artifact/repetition/Data/newSha.csv")
data$combined = data$combinedOriginal # does not matter
data$devs = data$committer # does not matter
data = summarizeByLanguage(data)
```

First, we download projects. This gets us the projects' metadata, all commits, and all unique files changed in the commits. Out of 728 projects, we downloaded 618 (causes: network failures during download, projects going private, or projects being deleted) and analyzed 513 (node.js, which we used to download the projects, segfaulted on several of them). The commits reported for the study were then analyzed; and for each project, we remember the list of commits used in the study.

`Total records:     1578165                                                    
Total projects:    729                                                        
Multi-commits:     46526                                                      
Unique commits:    1531639 
`

Multi-commits are commits that have multiple languages.

The downloaded projects were matched. Since the study has project names without repository owners, matches could be ambiguous.  We end up with 423 matched projects. One item, dogecoin, has the same name but two different projects. For each project, we looked at all commits and classified them as:

* valid (i.e. present in the study and in the project)
* irrelevant (i.e. present in the project, but not relevant to the study since they do not change any file in the studied languages)
* missing (present in the project, but not in the study, while changing at least one file in studied language)
* invalid (present in the study, not present in the project)

This data has been obtained by running the `commits-verifier` tool.

<!--%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%-->

<!--%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%-->

## Results on missing commits

<!--%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%-->

<!--%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%-->

First some verification and data cleanup -- if we matched a project wrongly, then we would see all invalid commits.

```r id=f768f3bc-7157-4bcb-90a5-70a2a06f4a9a
mc = read.csv("./input_data/missing-commits.csv")
mc %>% filter(invalid > 0) %>% arrange(desc(invalid))
```

[__out-0.csv][nextjournal#output#f768f3bc-7157-4bcb-90a5-70a2a06f4a9a#__out-0.csv]

There are four projects with invalid counts larger than 1.  In the case of "framework" and "hw", the numbers are large enough to be worth setting them aside. The case of "generator-mobile", where we have only invalid commits, suggests a badly matched project. We ignore it.  Lastly, "DefenitelyTyped" has 8 mistmatched commits -- but since it is one of the TypeScript projects that contains no code, we can safely ignore it.

```r id=8f178b20-135b-42ca-bec3-8f4b24051380
mc %>% filter(invalid <= 1) -> missing_commits

valid_sum <- sum(missing_commits$valid)
check(valid_sum == 426845)
missing_sum <- sum(missing_commits$missing)
check(missing_sum == 106411)
ratio_missing <- round(missing_sum/(missing_sum+valid_sum)*100,2)
check(ratio_missing == 19.95)
out("MissingCommitsThousands", round(missing_sum/1000,0))
out("MissingCommitsRatio", ratio_missing)
```

In total, we have seen 426k commits in the projects we have cross-checked. There were 106k missing commits (19.95%).

* The number of commits per project is skewed towards very few valid commits
* Invalid commits are in almost every project, and there are projects that are almost entirely missing

Projects with the highest ratio of missing commits:

```r id=01a644ba-0d1e-4a6c-ab9d-f6497f3ca20d
missing_commits %>% mutate(ratio = round(missing/(missing+valid)*100,2)) %>% arrange(desc(ratio))
```

[__out-0.csv][nextjournal#output#01a644ba-0d1e-4a6c-ab9d-f6497f3ca20d#__out-0.csv]

* V8 is high on the list -- the 12th most incomplete project (around 70% of commits are missing).

*`ggsave()` disabled due to a strange font error —MPD*

```r id=24164315-9939-4f8c-87da-ba4ea0706c2c
data %>% group_by(language) %>% summarize(commits = sum(commits)) -> commits_by_lang
commits_by_lang[commits_by_lang$language == "C", 2] <- commits_by_lang[commits_by_lang$language == "C", 2] +
                                                        commits_by_lang[commits_by_lang$language == "C++", 2]
commits_by_lang %>% filter(language != "C++") -> commits_by_lang
commits_by_lang %>% mutate(missing = 0) -> c
c$language <- as.character(c$language)

c[c$language == "C", 3] <- sum(missing_commits$cpp)  #C++ and C together
c[1, 1] <- "C/C++"
c[c$language == "C#",3] <-  sum(missing_commits$cs)
c[c$language == "Objective-C",3] <-  sum(missing_commits$objc)
c[c$language == "Go",3] <-  sum(missing_commits$go)
c[c$language == "Coffeescript",3] <-  sum(missing_commits$coffee)
c[c$language == "Javascript",3] <-  sum(missing_commits$js)
c[c$language == "Ruby",3] <-  sum(missing_commits$ruby)
c[c$language == "Typescript",3] <-  sum(missing_commits$ts)
c[c$language == "Php",3] <-  sum(missing_commits$php)
c[c$language == "Python",3] <-  sum(missing_commits$python)
c[c$language == "Perl",3] <-  sum(missing_commits$perl)
c[c$language == "Clojure",3] <-  sum(missing_commits$clojure)
c[c$language == "Erlang",3] <-  sum(missing_commits$erlang)
c[c$language == "Haskell",3] <-  sum(missing_commits$haskell)
c[c$language == "Scala",3] <-  sum(missing_commits$scala)
c[c$language == "Java",3] <-  sum(missing_commits$java)

c %>% mutate(ratio = round(missing/(commits+missing)*100,0)) %>% arrange(desc(ratio)) %>% as.data.frame -> ratio_missing

ggplot(data = ratio_missing, aes(x = reorder(language, ratio), y = ratio)) + 
    geom_bar(stat="identity") +
    xlab("") + ylab("Percentage missing commits") +
    annotate("text", x = "Perl", y = 20, label = paste(ratio_missing[ratio_missing$language=="Perl",4], "%", sep = ""), color = "white") +
    theme(axis.text.x = element_text(angle = 60, hjust = 1)) +
    coord_cartesian(ylim=c(0, 20)) -> p

#ggsave(paste(WORKING_DIR, "/Figures/ratio_missing.pdf", sep = ""), p, width=5, height=2, units="in", scale=1.5)
print(p)
out("PerlMissingRatio", ratio_missing[ratio_missing$language=="Perl",4])
```

![__out-0][nextjournal#output#24164315-9939-4f8c-87da-ba4ea0706c2c#__out-0]

Perl is the outlier here, then Erlang, Go, PHP, and JavaScript.

```r id=3f80cedb-d800-4133-a745-2b3ecf4d923a
remove(WORKING_DIR)
```

# Commits Survey (commit_survey.Rmd)

```r id=beb7a112-e588-4864-b261-ca0eb39d913f
source("implementation.R")
initializeEnvironment("./artifact/commit_survey")
```

## Get developers' labels into one combined dataframe

```r id=ab69e736-ea25-4208-af90-7cdc3de3befb
library(dplyr)
library("fst")
library("irr")

inputPath = "./input_data/commit_survey"
# read all files in a directory
answers = list.files(inputPath)
data = lapply(answers, function(x) { 
    cat(x, "\n")
    read.csv(paste(inputPath, x, sep = "/"), header = F, sep =",") 
    })
i = 1
while (i <= length(data)) {
    d = data[[i]]
    d$category = as.character(d$V2)
    d$category[is.na(d$category)] = 0
    d$category[d$category != 0] = 1
    data[[i]]$category = as.numeric(d$category)
    i = i + 1
}

# if this is set to T, pulls data from the extra developers we had available for the survey as well - but these finished later, or did not finish at all so we only used their subset. 
if (F) {
    inputPathExtra = "./input_data/commit_survey_extras"
    answersExtra = list.files(inputPathExtra)
    dataExtra = lapply(answersExtra, function(x) { read.csv(paste(inputPathExtra, x, sep = "/"), header = F, sep =",") })
    i = 1
    while (i <= length(dataExtra)) {
        d = dataExtra[[i]]
        d$category = as.character(d$V2)
        d$category[is.na(d$category)] = 0
        d$category[d$category != 0] = 1
        dataExtra[[i]]$category = as.numeric(d$category)
        i = i + 1
    }
    
    # join the single dataframe per file into a single dataframe with all of them
    
    
    data = rbind(data, dataExtra)

}
separateDevs = data
data = do.call("rbind", data)
# now aggregate according to the commit
#data$category = as.character(data$V2)
#data$category[is.na(data$category)] = 0
#data$category[data$category == "0"] = 0
#data$category[data$category != 0] = 1
data$category = as.numeric(data$category)
data$test = rep(1, nrow(data))
sanity_check = aggregate(test ~ V1, data, sum) # we expect the sanity check to be three for all commits
if(!all(sanity_check$test == 3)) cat("Error -- Not all commits have 3 votes\n")
dd = data
data = aggregate(category ~ V1, data, sum)
data$score = data$category / sanity_check$test # 3 # sanity_check$test
#data$score = sapply(data$category, function(x) mean(sapply(x, function (x) if ( x == 0) 0 else 1)))
data$label = ifelse(data$score>=0.5, 1, 0) # This will never == 0.5 because we will have 3 votes
significant <- data[sapply(data$category, length) > 1,]
discord <- significant[(significant$score != 0) & (significant$score != 1), ]
```

## Compare developers' labels to study's labels

```r id=aa71794f-8d95-40ac-898e-de10fe5f6388

stripGHUrl = function(what) {
    cat(what)
    substr(what, nchar(what) - 39, nchar(what))
}
ourLabelsPath <- "./input_data/petrs_commits"
buggycommits <- read.csv(paste(ourLabelsPath, "buggy_commits.csv", sep="/"), header=F)
buggycommits$V1 <- as.character(buggycommits$V1)
#buggycommits$V1 = sapply(buggycommits$V1, stripGHUrl)
buggycommits$ours = buggycommits$V3
buggycommits$paper = 1
nonbuggycommits <- read.csv(paste(ourLabelsPath, "non_buggy_commits.csv", sep="/"), header=F)
nonbuggycommits$V1 <- as.character(nonbuggycommits$V1)
#nonbuggycommits$V1 = sapply(nonbuggycommits$V1, stripGHUrl)
nonbuggycommits$ours = as.numeric(!nonbuggycommits$V3)
nonbuggycommits$paper = 0
allcommits = rbind(buggycommits, nonbuggycommits)

data$V1 = as.character(data$V1)
allcommits$V2 = as.character(allcommits$V2)
```

```r id=d6032ea4-b6b7-456c-876b-8030e1d994d2
# Calculate disagreement

getPaperLabels <- function() {
    labels = rep(0, nrow(data))
    for (r in 1:nrow(data)) {
        df2row = allcommits[allcommits$V1 == data$V1[[r]],]
        labels[r] = df2row$paper;
    }
    labels    
}

getOurLabels <- function() {
    labels = rep(0, nrow(data))
    for (r in 1:nrow(data)) {
        df2row = allcommits[allcommits$V1 == data$V1[[r]],]
        labels[r] = df2row$ours;
    }
    labels    
}

data$ours = getOurLabels()
data$paper = getPaperLabels()

data$agreePaper = as.numeric(data$label == data$paper)
data$agreeOurs = as.numeric(data$label == data$ours)

#data$disagreeOurs = getDisagreementLabels(data, allcommits)
#data$disagreePaper = getDisagreementLabelsPaper(data, allcommits)

write.csv(data, file=paste0(WORKING_DIR, "/Data/commit_survey_results.csv"))
```

```r id=5b1a8532-4eee-4521-940d-8cf5c3a122d7
falsePositives <- nrow(data[data$paper == 1 & data$label == 0,])
falseNegatives <- nrow(data[data$paper == 0 & data$label == 1,])

out("commitsFalsePositives",  paste0(round(falsePositives / nrow(buggycommits) * 100, 1),"\\%"))
out("commitsFalseNegatives", paste0(round(falseNegatives / nrow(nonbuggycommits) * 100, 1), "\\%"))

#cat(paste("Percentage of false positives (Paper vs Devs): ", falsePositives / nrow(buggycommits), "\n"))
#cat(paste("Percentage of false negatives: (Paper vs Devs)", falseNegatives / nrow(nonbuggycommits), "\n\n"))
```

```r id=3cb01fd1-9e27-41ae-ad98-9b08f12fb2d0
unanimous_falsePositives <- nrow(data[data$paper == 1 & data$score == 0,])
out("commitsUnanimousFalsePositives", paste0(round(unanimous_falsePositives / falsePositives * 100, 1), "\\%"))
#cat("Percentage of false positives upon which developers unanimously agreed: ", unanimous_falsePositives / falsePositives)
```

```r id=daf873d3-bf4f-418e-8f75-2a3ff354b954
PfalsePositives <- nrow(data[data$paper == 1 & data$ours == 0,])
PfalseNegatives <- nrow(data[data$paper == 0 & data$ours == 1,])

cat(paste("Percentage of false positives (Paper vs Us): ", PfalsePositives / nrow(buggycommits), "\n"))
cat(paste("Percentage of false negatives (Paper vs Us): ", PfalseNegatives / nrow(nonbuggycommits), "\n\n"))
```

```r id=046b188f-b4e4-4691-8486-61d40db181d4
PDfalsePositives <- nrow(data[data$label == 1 & data$ours == 0,])
PDfalseNegatives <- nrow(data[data$label == 0 & data$ours == 1,])

cat(paste("Percentage of false positives (Devs vs Us): ", PDfalsePositives / nrow(data[data$label == 1,]), "\n"))
cat(paste("Percentage of false negatives (Devs vs Us): ", PDfalseNegatives / nrow(data[data$label == 0,]), "\n"))
```

Let's look at how certain people are:

```r id=9ff6b827-7237-4f22-b5d4-0d7af9de2c53
cat(paste("3 votes not a bug:                 ", length(data$score[data$score == 0]), "\n"))
cat(paste("Likely not a bug (1 vote for bug): ", length(data$score[data$score == 1/3]), "\n"))
cat(paste("Likely a bug (2 votes for bug):    ", length(data$score[data$score == 2/3]), "\n"))
cat(paste("3 votes for a bug:                 ", length(data$score[data$score == 1]), "\n"))
```

This is cool, but let's do this for buggy and non-buggy commits as determined by the paper:

```r id=70f27289-ae00-4623-89e4-0536127492f8
buggy = data[data$paper == 1,]
cat("Paper labelled as buggy\n")
cat(paste("3 votes not a bug:                 ", length(buggy$score[buggy$score == 0]), "\n"))
cat(paste("Likely not a bug (1 vote for bug): ", length(buggy$score[buggy$score == 1/3]), "\n"))
cat(paste("Likely a bug (2 votes for bug):    ", length(buggy$score[buggy$score == 2/3]), "\n"))
cat(paste("3 votes for a bug:                 ", length(buggy$score[buggy$score == 1]), "\n"))
cat("Paper labelled as non-buggy\n")
nonbuggy = data[data$paper == 0,]
cat(paste("3 votes not a bug:                 ", length(nonbuggy$score[nonbuggy$score == 0]), "\n"))
cat(paste("Likely not a bug (1 vote for bug): ", length(nonbuggy$score[nonbuggy$score == 1/3]), "\n"))
cat(paste("Likely a bug (2 votes for bug):    ", length(nonbuggy$score[nonbuggy$score == 2/3]), "\n"))
cat(paste("3 votes for a bug:                 ", length(nonbuggy$score[nonbuggy$score == 1]), "\n"))
```

And now with me

```r id=34d7ce65-9228-44c2-9a45-36d89c2f10fe
buggyp = data[data$ours == 1,]
cat("Peta labelled as buggy\n")
cat(paste("3 votes not a bug:                 ", length(buggyp$score[buggyp$score == 0]), "\n"))
cat(paste("Likely not a bug (1 vote for bug): ", length(buggyp$score[buggyp$score == 1/3]), "\n"))
cat(paste("Likely a bug (2 votes for bug):    ", length(buggyp$score[buggyp$score == 2/3]), "\n"))
cat(paste("3 votes for a bug:                 ", length(buggyp$score[buggyp$score == 1]), "\n"))
cat("Peta labelled as non-buggy\n")
nonbuggyp = data[data$ours == 0,]
cat(paste("3 votes not a bug:                 ", length(nonbuggyp$score[nonbuggyp$score == 0]), "\n"))
cat(paste("Likely not a bug (1 vote for bug): ", length(nonbuggyp$score[nonbuggyp$score == 1/3]), "\n"))
cat(paste("Likely a bug (2 votes for bug):    ", length(nonbuggyp$score[nonbuggyp$score == 2/3]), "\n"))
cat(paste("3 votes for a bug:                 ", length(nonbuggyp$score[nonbuggyp$score == 1]), "\n"))
```

Now look at the developers and calculate their ratio of bugs found:

```r id=60eb97d6-b9f8-48c3-8901-294edfa4893c
for (d in separateDevs) {
    cat(sum(as.numeric(d$category)) / length(d$category), "\n")
}
```

Ok, what we can do, is to compute for each developer how often he is in opposition to the rest:

```r id=9a35a691-e515-4921-a093-caf7b5d16d8a
devOps <- function(index) {
    score = data$score
    names(score) = data$V1
    d = separateDevs[[index]]
    dc = d$category
    names(dc) = d$V1
    bugX = 0
    nonBugX = 0
    resBug = 0
    resNonBug = 0
    for (i in d$V1) {
        s = score[[i]]
        if (dc[[i]] == 0) {
            if (s == 2/3)
                nonBugX = nonBugX + 1
            else if (s == 1/3)
                resNonBug = resNonBug + 1
        } else {
            if (s == 1/3)
                bugX = bugX + 1
            else if (s == 2/3)
                resBug = resBug + 1
        }
    }
    c(bugX, nonBugX, resBug, resNonBug) / length(d$V1)
}
for (i in 1:10)
    print(devOps(i))
```

## Calculating Cohen's Kappa

```r id=59edef53-67a8-4ad0-b212-6de2875c8ed4
# the number of people in the survey
numDevs = length(separateDevs)
# now calculate a dataframe where we have aggregated commits and for each commit we have the category assigned by all of the developers (NA if the developer did not see the commit)
ratings = NULL
tmp = list()
for (n in (1:numDevs)) {
    x = cbind(as.character(separateDevs[[n]]$V1), separateDevs[[n]]$category)
    colnames(x) = c("url", paste0("x",n))
    if (n == 1) {
        ratings = x
    } else {
        ratings = merge(ratings, x, by = c("url"), all = T)
    }
}
# create matrix of only the ratings w/o the commit urls
#ratings_matrix = as.matrix(subset(ratings, select = - c(url)))
```

Now that we have the matrix, we should calculate the Cohen's Kappa to determine the interrater agreement between the raters. Because Cohen's Kappe is only defined for 2 raters, we use the Light's Kappa, which is an average over all pairwise Cohen Kappas. The fact that different commits were reviewed by different reviewers further complicates this. We therefore create a submatrix for each 2 pairs of developers, remove any NA rows, calculate the kappas and then report the results and distribution:

```r id=7edb7013-8ef8-4467-bc9f-f1167d0b87ed
kvalue = c()
kpvalue = c()
ksize = c()
kfirst = c()
ksecond = c()
n = 1
for (first in (1:(numDevs - 1))) {
    for (second in ((first + 1):numDevs)) {
        tmp = cbind(ratings[colnames(ratings)[[first + 1]]], ratings[colnames(ratings)[[second + 1]]])
        tmp = na.omit(tmp)
        if (nrow(tmp) != 0) {
            k = kappa2(tmp)
            kvalue[[n]] = k$value
            kpvalue[[n]] = k$p.value
            ksize[[n]] = nrow(tmp)
            kfirst[[n]] = first
            ksecond[[n]] = second
            n = n + 1
        } 
    }
}
summary(kvalue)
summary(kpvalue)
summary(ksize)
out("commitsKappaMedian", round(median(kvalue), 3))
out("commitsKappaMin", round(min(kvalue), 3))
out("commitsKappaMax", round(max(kvalue), 3))
out("commitsKappaPValMedian", round(median(kpvalue), 3))
out("commitsKappaPValThird", round(quantile(kpvalue, 0.75), 3))
```

Ok, all of our kappas are positive. Most of the p-values are also very small.

```r id=0653cb27-08db-4c48-b5ff-4970ea027a34
boxplot(kvalue)
boxplot(kpvalue)
sort(kpvalue)
```

![__out-0][nextjournal#output#0653cb27-08db-4c48-b5ff-4970ea027a34#__out-0]

![__out-1][nextjournal#output#0653cb27-08db-4c48-b5ff-4970ea027a34#__out-1]

# Threats To Validity (threats_to_validity.Rmd)

```r id=2f0ff0c8-78dc-47ba-9aca-8eda9a3d3372
# load the file containing the actual implementation details
knitr::opts_chunk$set(echo = FALSE)
source("implementation.R")
initializeEnvironment("./artifact/threats-to-validity")
```

## Tests in the files

```r id=b84a6c4b-3252-449e-a5f6-595dbe485166
data = read.csv("./artifact/repetition/Data/newSha.csv")
files = loadEverything()
```

Let's see how many of the files are tests. First clear the files dataset to only contain commits that are in the `data` we analyze:

```r id=9d946af6-e55e-41b8-acd7-841798dbf451
files = files %>% filter(sha %in% unique(data$sha))
```

```r id=b44f3d02-19d1-4054-b249-4045eb3cc688
files %>% filter(str_detect(file_name,"Test|test")) -> tests
out("ratioTestsFiles",round(nrow(tests)/nrow(files)*100,1))
files %>% filter(!str_detect(file_name,"Test|test")) -> notests
out("testFilesCommitted", nrow(tests))
out("ratioTestFilesCommittedOverAll", round(nrow(tests)/nrow(files)*100,1))
```

# Appendix

Creating the environment for the analysis.  System dependencies.

```bash id=dd493f53-2ad1-4176-8b00-15ef8bbbf1ea
apt-get -qq update
DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends \
  git-lfs pandoc tcsh
apt-get clean
rm -r /var/lib/apt/lists/* # Clear package list so it isn't stale
```

Clone the repo to get `.R` files and data.

```bash id=77f29bb3-dc3f-4f33-80c8-8b2e45ed5010
git lfs clone https://github.com/PRL-PRG/TOPLAS19_Artifact toplas
mv toplas/* ./
rm -r toplas
```

And install R packages.

```r id=5fc2098e-f67d-4885-9a29-f62eb73f9d6e
list.of.packages <- c("rmarkdown","car","data.table","assertthat","fst",
                      "stringr", "plyr","ggplot2","ggrepel","scales","xtable",
                      "MASS","gplots","dplyr","gridExtra","grid","tidyr","irr")
new.packages <- list.of.packages[
  !(list.of.packages %in% installed.packages()[,"Package"])]
if(length(new.packages)) install.packages(new.packages, Ncpus=4)
```

[nextjournal#output#c6924612-47e4-40cd-94e3-56c8f1bd6871#__out-0.csv]:
<https://nextjournal.com/data/QmbqnrGkp3GsNea43iDC1FcoVv5xMaudpEhMwEzPzrm4CD?content-type=text/csv&node-id=c6924612-47e4-40cd-94e3-56c8f1bd6871&filename=__out-0.csv&node-kind=output>

[nextjournal#output#fb2319e2-1d3e-46da-86c7-2975bb5365b0#__out-0.csv]:
<https://nextjournal.com/data/QmNc7pi86jTPtpoAE7amNZ4NmcG9UJgQqQ7yR83a915vNR?content-type=text/csv&node-id=fb2319e2-1d3e-46da-86c7-2975bb5365b0&filename=__out-0.csv&node-kind=output>

[nextjournal#output#1cf447fc-2b26-4153-94fd-3d5bd3fcab10#__out-0.csv]:
<https://nextjournal.com/data/QmXSim2okBtLNBwk9jRSaUANoz8mLPvFmzUrZTgJi58cak?content-type=text/csv&node-id=1cf447fc-2b26-4153-94fd-3d5bd3fcab10&filename=__out-0.csv&node-kind=output>

[nextjournal#output#aa6ebef6-36b2-4aa9-87fc-9bc4b5f2422d#__out-0.csv]:
<https://nextjournal.com/data/Qme3X5ERgg78DqNm1R6PTqJU5QvARrLfzou38NCpTiDXro?content-type=text/csv&node-id=aa6ebef6-36b2-4aa9-87fc-9bc4b5f2422d&filename=__out-0.csv&node-kind=output>

[nextjournal#output#c384c48a-7e1a-4362-bb5e-9fc84d0472f2#__out-0.csv]:
<https://nextjournal.com/data/QmZJhhjTA69MUwZ5MXUywXrra6hrUUdTBH1AeSbS3UU8Ae?content-type=text/csv&node-id=c384c48a-7e1a-4362-bb5e-9fc84d0472f2&filename=__out-0.csv&node-kind=output>

[nextjournal#output#ea87e534-201f-4a02-9864-afa527d5e926#__out-0.csv]:
<https://nextjournal.com/data/QmUrURhHLqahu8bzX3gzwCpjTNCnnoZJnyyGcSWS2aZTWQ?content-type=text/csv&node-id=ea87e534-201f-4a02-9864-afa527d5e926&filename=__out-0.csv&node-kind=output>

[nextjournal#output#cdd13d43-6b99-45b8-8c14-d50d1d6d481a#__out-0.csv]:
<https://nextjournal.com/data/QmPPdzqekq45Q8yaidPdpLKVJ2NMACLGTkK8qZMXQ2WUmE?content-type=text/csv&node-id=cdd13d43-6b99-45b8-8c14-d50d1d6d481a&filename=__out-0.csv&node-kind=output>

[nextjournal#output#2bdf86cd-0f2b-4fab-bd36-87261d6fe180#__out-0.csv]:
<https://nextjournal.com/data/QmNSCESkvyo1MjioEYYDQNCi2rm47mtBommrLxCwQkz31v?content-type=text/csv&node-id=2bdf86cd-0f2b-4fab-bd36-87261d6fe180&filename=__out-0.csv&node-kind=output>

[nextjournal#output#6920dcff-21c0-48c7-934d-46696eb94a3f#__out-0.csv]:
<https://nextjournal.com/data/QmZGtxBQb7hddi9jm8Ui2W8a7Vbh4yh3E4VWdK5j96zAwu?content-type=text/csv&node-id=6920dcff-21c0-48c7-934d-46696eb94a3f&filename=__out-0.csv&node-kind=output>

[nextjournal#output#6df945d4-528f-4437-80f6-21bd47f22da9#__out-0.csv]:
<https://nextjournal.com/data/QmWgmEFpQe4HbKwvHHBtzwF64y363nfoVbFDEbtD51aZe7?content-type=text/csv&node-id=6df945d4-528f-4437-80f6-21bd47f22da9&filename=__out-0.csv&node-kind=output>

[nextjournal#output#3d0f85f3-894d-46b0-8d0d-b8cea873fd95#__out-0.csv]:
<https://nextjournal.com/data/QmaXJnGLKaketjbNcGexe4DtMHiLpgDfr3PaTFBsgQ4PUC?content-type=text/csv&node-id=3d0f85f3-894d-46b0-8d0d-b8cea873fd95&filename=__out-0.csv&node-kind=output>

[nextjournal#output#7ab16acc-7b3f-4be8-8719-3bf561728667#__out-0]:
<https://nextjournal.com/data/QmT95mwunTUGjztuo2WJWAWWvDujwZRdGr5WbHDL8E5tuT?content-type=image/svg%2Bxml&node-id=7ab16acc-7b3f-4be8-8719-3bf561728667&filename=__out-0&node-kind=output>

[nextjournal#output#e6e68ace-7464-4511-8c9d-57109c202ade#__out-0]:
<https://nextjournal.com/data/QmNji1UtkNsfYaWJtHAFsSdHcn7WKjWuP4P5DAiE1qEFvh?content-type=image/svg%2Bxml&node-id=e6e68ace-7464-4511-8c9d-57109c202ade&filename=__out-0&node-kind=output>

[nextjournal#output#be6f7148-73c9-40ae-bc72-6514b7cbe4b3#__out-0.csv]:
<https://nextjournal.com/data/QmXwNiZft8NjVwktwu9gD4fsAWLaw25HEssbgMQmQUbTsz?content-type=text/csv&node-id=be6f7148-73c9-40ae-bc72-6514b7cbe4b3&filename=__out-0.csv&node-kind=output>

[nextjournal#output#b311cf2a-351f-4645-a50f-49c3d2461c1b#__out-0.csv]:
<https://nextjournal.com/data/QmU7fDXNXvB1KY4KSEKHNU4xnGFLP8ESXjQQbT9Cuwmspm?content-type=text/csv&node-id=b311cf2a-351f-4645-a50f-49c3d2461c1b&filename=__out-0.csv&node-kind=output>

[nextjournal#output#e835d074-edc9-40d2-a974-4e598b2a7810#__out-0]:
<https://nextjournal.com/data/Qmd3BMWh8m5WapKrCHyNYsvxHjA5JZQM5iafs3KmCZLyuk?content-type=image/svg%2Bxml&node-id=e835d074-edc9-40d2-a974-4e598b2a7810&filename=__out-0&node-kind=output>

[nextjournal#output#490f0c1f-fd45-4847-9a31-3cc789f2b9b8#__out-0]:
<https://nextjournal.com/data/Qmau8xBneHExVtHvFxf3DaMoKDLErdY7mD2qNWFTxuYCV5?content-type=image/svg%2Bxml&node-id=490f0c1f-fd45-4847-9a31-3cc789f2b9b8&filename=__out-0&node-kind=output>

[nextjournal#output#0034f7d6-85c5-4f6d-b099-4ae229f88814#__out-0]:
<https://nextjournal.com/data/QmRzyunE84sNHoJ5SqYbb3HBZpRW7bihUhTnftpHi6bj5b?content-type=image/svg%2Bxml&node-id=0034f7d6-85c5-4f6d-b099-4ae229f88814&filename=__out-0&node-kind=output>

[nextjournal#output#497f38e6-82bf-44d4-90c5-4e25ba2fb8bd#__out-1]:
<https://nextjournal.com/data/QmT9zTTPBYhK4tQZC8uAurRTwXdQwEvmoH5bjGrT8Rxb7a?content-type=image/svg%2Bxml&node-id=497f38e6-82bf-44d4-90c5-4e25ba2fb8bd&filename=__out-1&node-kind=output>

[nextjournal#output#497f38e6-82bf-44d4-90c5-4e25ba2fb8bd#__out-0.csv]:
<https://nextjournal.com/data/QmYwRWHH12JR32VUjW757SBBYCY6aaB2QnHrjeoLqiamkm?content-type=text/csv&node-id=497f38e6-82bf-44d4-90c5-4e25ba2fb8bd&filename=__out-0.csv&node-kind=output>

[nextjournal#output#a69c52bc-3752-4fb0-8996-682e07f4afab#__out-0]:
<https://nextjournal.com/data/QmRceVwzxowgPhKKp6v4afjJVVSmoZUmRKLRxk8oGpB9K6?content-type=image/svg%2Bxml&node-id=a69c52bc-3752-4fb0-8996-682e07f4afab&filename=__out-0&node-kind=output>

[nextjournal#output#a65ca8e7-abca-4c4d-bef4-355324507c99#__out-0]:
<https://nextjournal.com/data/QmUj5DCZaC3DZ1i9wqNSrR53XjijXw8gJowEaBa2wzS6b4?content-type=image/svg%2Bxml&node-id=a65ca8e7-abca-4c4d-bef4-355324507c99&filename=__out-0&node-kind=output>

[nextjournal#output#73aceb78-24b4-4c9d-bad7-01cf035d3d99#__out-2]:
<https://nextjournal.com/data/QmeaTADppCPwwJKm26f7u53cL8TEcu7noGw1CwoDesAjkE?content-type=image/svg%2Bxml&node-id=73aceb78-24b4-4c9d-bad7-01cf035d3d99&filename=__out-2&node-kind=output>

[nextjournal#output#73aceb78-24b4-4c9d-bad7-01cf035d3d99#__out-0]:
<https://nextjournal.com/data/QmbSawECUi3HSD85aavrXco49wnLRVNQDoHKHXvm3g4qDx?content-type=image/svg%2Bxml&node-id=73aceb78-24b4-4c9d-bad7-01cf035d3d99&filename=__out-0&node-kind=output>

[nextjournal#output#73aceb78-24b4-4c9d-bad7-01cf035d3d99#__out-1]:
<https://nextjournal.com/data/QmVyfG3vS9cLtLECTaPg6cQMmam9QTQ6FmhkoPPbLJNDyQ?content-type=image/svg%2Bxml&node-id=73aceb78-24b4-4c9d-bad7-01cf035d3d99&filename=__out-1&node-kind=output>

[nextjournal#output#46e51b59-b929-4539-b343-81af11447fe5#__out-0.csv]:
<https://nextjournal.com/data/Qmf16U9zES6BQ8NHA3SzJFUdfevVRjMZfB8ZFBwz7HAnyD?content-type=text/csv&node-id=46e51b59-b929-4539-b343-81af11447fe5&filename=__out-0.csv&node-kind=output>

[nextjournal#output#f35490a1-51cd-4200-96ab-1dc14d265847#__out-0.csv]:
<https://nextjournal.com/data/Qmcr7FvZR9XbqiminwaFTBkCGRwun4cjEHVy7CUj2y89Yy?content-type=text/csv&node-id=f35490a1-51cd-4200-96ab-1dc14d265847&filename=__out-0.csv&node-kind=output>

[nextjournal#output#24514db5-6b38-4647-ae6e-6af8dd632f6a#__out-0.csv]:
<https://nextjournal.com/data/Qmf125cra7qRqaLcFPo4pJz8dPTV1YiXZdkynT2qm7Z7fp?content-type=text/csv&node-id=24514db5-6b38-4647-ae6e-6af8dd632f6a&filename=__out-0.csv&node-kind=output>

[nextjournal#output#fe2273fd-38ef-443a-b042-e3ad1045e151#__out-0.csv]:
<https://nextjournal.com/data/QmVyjC2HVHAXSMWhPa4CHrP9aNRGCUFnBxr5Ke9vErJkgN?content-type=text/csv&node-id=fe2273fd-38ef-443a-b042-e3ad1045e151&filename=__out-0.csv&node-kind=output>

[nextjournal#output#0b4d60f3-b305-40ee-ac43-9f90d12a057b#__out-0.csv]:
<https://nextjournal.com/data/QmVNyCjibtPFEVEjb4e4N69wnZ9PuPWEuyBgvSkP4cBo5b?content-type=text/csv&node-id=0b4d60f3-b305-40ee-ac43-9f90d12a057b&filename=__out-0.csv&node-kind=output>

[nextjournal#output#430c34d0-686b-4d21-a864-83e02ebb246f#__out-0.csv]:
<https://nextjournal.com/data/QmTeM8UiyzbReVevtt9NPCXD5U3wNuL2NeCyL5hAtaMR4d?content-type=text/csv&node-id=430c34d0-686b-4d21-a864-83e02ebb246f&filename=__out-0.csv&node-kind=output>

[nextjournal#output#b09318ab-6250-4c72-9f55-42429737228b#__out-0]:
<https://nextjournal.com/data/QmNq73q9iccU7zJZC8LxVkyN5hMyzrzLF27WNk3jBVWjX2?content-type=image/svg%2Bxml&node-id=b09318ab-6250-4c72-9f55-42429737228b&filename=__out-0&node-kind=output>

[nextjournal#output#52c0b24f-3cd6-4ca4-ba41-2f000563abab#__out-0.csv]:
<https://nextjournal.com/data/QmRUgt8pi7A5VjxLHjPN1ZoEEwJNRvSgqmVfDf85rHYEQj?content-type=text/csv&node-id=52c0b24f-3cd6-4ca4-ba41-2f000563abab&filename=__out-0.csv&node-kind=output>

[nextjournal#output#4403efe0-1c1d-4faf-935b-c111c7644a3f#__out-0.csv]:
<https://nextjournal.com/data/QmSMXtmFiF2AjdqgfSXdp8v2YmRf9BL1vNsWV4R7FkYRjN?content-type=text/csv&node-id=4403efe0-1c1d-4faf-935b-c111c7644a3f&filename=__out-0.csv&node-kind=output>

[nextjournal#output#50218144-c124-476a-9c81-c78f4eed67fd#__out-0.svg]:
<https://nextjournal.com/data/QmbL4zKH7wtySfoa3pcfMyyw1b5xepKNx7oRzjdwemaeGJ?content-type=image/svg%2Bxml&node-id=50218144-c124-476a-9c81-c78f4eed67fd&filename=__out-0.svg&node-kind=output>

[nextjournal#output#a6bce8cd-c677-41e6-a4a5-b08666417620#__out-0.csv]:
<https://nextjournal.com/data/QmPEK1dS5CSpA5fzbdUWAnY26R188quwMgPtrNCHBCnxSr?content-type=text/csv&node-id=a6bce8cd-c677-41e6-a4a5-b08666417620&filename=__out-0.csv&node-kind=output>

[nextjournal#output#a6bce8cd-c677-41e6-a4a5-b08666417620#__out-1.svg]:
<https://nextjournal.com/data/QmWVq9W9S5aHFeFS3VM3PSmazYeUTVSMd3WRht3N8yaFdF?content-type=image/svg%2Bxml&node-id=a6bce8cd-c677-41e6-a4a5-b08666417620&filename=__out-1.svg&node-kind=output>

[nextjournal#output#1e7dc2c5-fe49-49d5-a68e-809e405860bd#__out-0.csv]:
<https://nextjournal.com/data/Qmc4A1MZNpHeQmagzhYfqQX4Ttqgac1hq3BYW4Sm7xvn7u?content-type=text/csv&node-id=1e7dc2c5-fe49-49d5-a68e-809e405860bd&filename=__out-0.csv&node-kind=output>

[nextjournal#output#8b81d611-da90-4062-9a3c-8e29e3a300ee#__out-0.csv]:
<https://nextjournal.com/data/QmZo5hfnDuLsRCv6gKpwqTzTEfwtMa49to2sRbNrtqmTCf?content-type=text/csv&node-id=8b81d611-da90-4062-9a3c-8e29e3a300ee&filename=__out-0.csv&node-kind=output>

[nextjournal#output#f768f3bc-7157-4bcb-90a5-70a2a06f4a9a#__out-0.csv]:
<https://nextjournal.com/data/QmTUj8BNgZ3A2QLie7MzCDjNfj5Jvn4tWZqtG7Mpi3efmi?content-type=text/csv&node-id=f768f3bc-7157-4bcb-90a5-70a2a06f4a9a&filename=__out-0.csv&node-kind=output>

[nextjournal#output#01a644ba-0d1e-4a6c-ab9d-f6497f3ca20d#__out-0.csv]:
<https://nextjournal.com/data/QmQjbJbrkJjBM49Tt3QaS2TnZ4gerBBHpumeKNxerRrLzT?content-type=text/csv&node-id=01a644ba-0d1e-4a6c-ab9d-f6497f3ca20d&filename=__out-0.csv&node-kind=output>

[nextjournal#output#24164315-9939-4f8c-87da-ba4ea0706c2c#__out-0]:
<https://nextjournal.com/data/QmR6niYcQDXGLEgRFd8qU86ZqbUZgDXDud3ALVa7H8Ny1o?content-type=image/svg%2Bxml&node-id=24164315-9939-4f8c-87da-ba4ea0706c2c&filename=__out-0&node-kind=output>

[nextjournal#output#0653cb27-08db-4c48-b5ff-4970ea027a34#__out-0]:
<https://nextjournal.com/data/QmXAFPaTFvjiv2vxnTynZmnAjXLA9wo72b4RM6CM3ATcPY?content-type=image/svg%2Bxml&node-id=0653cb27-08db-4c48-b5ff-4970ea027a34&filename=__out-0&node-kind=output>

[nextjournal#output#0653cb27-08db-4c48-b5ff-4970ea027a34#__out-1]:
<https://nextjournal.com/data/QmVynL5Wn5xceeSFP7CrjZcMUrx5WQJDukPc7tjrdW5158?content-type=image/svg%2Bxml&node-id=0653cb27-08db-4c48-b5ff-4970ea027a34&filename=__out-1&node-kind=output>

<details id="com.nextjournal.article">
<summary>This notebook was exported from <a href="https://nextjournal.com/a/LWEWvgCnUSSB2MsccrixT?change-id=Cm2edMoY7MTBBVSi2jq4Dj">https://nextjournal.com/a/LWEWvgCnUSSB2MsccrixT?change-id=Cm2edMoY7MTBBVSi2jq4Dj</a></summary>

```edn nextjournal-metadata
{:article
 {:settings {:subtitle? true, :authors? true, :numbered? true},
  :nodes
  {"0034f7d6-85c5-4f6d-b099-4ae229f88814"
   {:compute-ref #uuid "b169d4e1-cacf-4259-9621-b91b96e7be83",
    :exec-duration 2217,
    :id "0034f7d6-85c5-4f6d-b099-4ae229f88814",
    :kind "code",
    :output-log-lines {:stdout 2},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "01a644ba-0d1e-4a6c-ab9d-f6497f3ca20d"
   {:compute-ref #uuid "ff5413e4-8be6-4b80-80d8-f5fe7af371e9",
    :exec-duration 178,
    :id "01a644ba-0d1e-4a6c-ab9d-f6497f3ca20d",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "046b188f-b4e4-4691-8486-61d40db181d4"
   {:compute-ref #uuid "6a414acb-04da-42e5-8a46-ebea0df2530f",
    :exec-duration 423,
    :id "046b188f-b4e4-4691-8486-61d40db181d4",
    :kind "code",
    :output-log-lines {:stdout 2},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "0597ff23-6b63-4368-a49e-a9d3ef5eab7e"
   {:compute-ref #uuid "8dc9de65-f30b-4cdf-b312-a099dd4d52e5",
    :exec-duration 397,
    :id "0597ff23-6b63-4368-a49e-a9d3ef5eab7e",
    :kind "code",
    :output-log-lines {:stdout 33},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "0653cb27-08db-4c48-b5ff-4970ea027a34"
   {:compute-ref #uuid "50524aea-e6a0-47c0-8e50-e7f96496426c",
    :exec-duration 443,
    :id "0653cb27-08db-4c48-b5ff-4970ea027a34",
    :kind "code",
    :output-log-lines {:stdout 9},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "06ea26de-f567-4f31-ad0a-4bf50fdedb84"
   {:compute-ref #uuid "ef1e9b4e-323f-4ffd-902f-353db024e54f",
    :exec-duration 2024,
    :id "06ea26de-f567-4f31-ad0a-4bf50fdedb84",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "07aff476-f04b-404c-8703-f40bf940cd41"
   {:compute-ref #uuid "844d2452-9988-418c-bbeb-60d2e7c0eb4c",
    :exec-duration 436,
    :id "07aff476-f04b-404c-8703-f40bf940cd41",
    :kind "code",
    :output-log-lines {:stdout 1},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "08d05a34-4457-42c5-b8e3-4f4387445335"
   {:compute-ref #uuid "a97df9ee-8917-4a74-80c4-d90d3334e517",
    :exec-duration 1474,
    :id "08d05a34-4457-42c5-b8e3-4f4387445335",
    :kind "code",
    :output-log-lines {:stdout 3},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "0992f2ec-40e7-4ac2-b363-49afa188b1b2"
   {:compute-ref #uuid "3cf7e7d3-a515-41ae-a39a-c043542f02d4",
    :exec-duration 8194,
    :id "0992f2ec-40e7-4ac2-b363-49afa188b1b2",
    :kind "code",
    :output-log-lines {:stdout 6},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "0b4d60f3-b305-40ee-ac43-9f90d12a057b"
   {:compute-ref #uuid "9bd5374c-adc4-4c36-bb50-2aae9425b0de",
    :exec-duration 221,
    :id "0b4d60f3-b305-40ee-ac43-9f90d12a057b",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "12cd028f-5fde-4c4f-af61-efc94a84b836"
   {:compute-ref #uuid "310c3225-febc-4678-8286-362e743f1502",
    :exec-duration 701,
    :id "12cd028f-5fde-4c4f-af61-efc94a84b836",
    :kind "code",
    :output-log-lines {:stdout 1},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "15e63d41-ea4a-4ab3-bca4-1e3d770ece91"
   {:compute-ref #uuid "219d5d86-4108-4106-b64e-e661f8ee81bf",
    :exec-duration 236,
    :id "15e63d41-ea4a-4ab3-bca4-1e3d770ece91",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "15f12eda-f44b-42ee-be20-5e76df2209c4"
   {:compute-ref #uuid "c3ea6fc0-6f48-4e84-8670-4a9b073dec0e",
    :exec-duration 1169,
    :id "15f12eda-f44b-42ee-be20-5e76df2209c4",
    :kind "code",
    :output-log-lines {:stdout 2},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "17374c10-375e-4d3c-8ce7-2b8a6e400923"
   {:compute-ref #uuid "32e18003-b613-47cf-8f08-16dfb27c8bd9",
    :exec-duration 590,
    :id "17374c10-375e-4d3c-8ce7-2b8a6e400923",
    :kind "code",
    :output-log-lines {:stdout 2},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "19fbed36-4418-4eaf-aa6a-95df78707aff"
   {:compute-ref #uuid "a7cab2fc-b680-4635-82b5-cf743fb8fefc",
    :exec-duration 188,
    :id "19fbed36-4418-4eaf-aa6a-95df78707aff",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "1cf447fc-2b26-4153-94fd-3d5bd3fcab10"
   {:compute-ref #uuid "5a8b8d3c-32d2-4b0f-a7e4-77054a63d163",
    :exec-duration 226,
    :id "1cf447fc-2b26-4153-94fd-3d5bd3fcab10",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "1da1c5f5-1d5a-453e-816b-27650a4f5477"
   {:compute-ref #uuid "1c8da3c1-320e-4179-bf6c-b081d26bebed",
    :exec-duration 679,
    :id "1da1c5f5-1d5a-453e-816b-27650a4f5477",
    :kind "code",
    :output-log-lines {:stdout 7},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "1e7dc2c5-fe49-49d5-a68e-809e405860bd"
   {:compute-ref #uuid "52cee9e7-667b-4ad4-944c-3eb106c0ffc9",
    :exec-duration 21479,
    :id "1e7dc2c5-fe49-49d5-a68e-809e405860bd",
    :kind "code",
    :output-log-lines {:stdout 9},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "203f844b-df5e-41b8-b2b7-28887630762d"
   {:compute-ref #uuid "3f7c8999-774c-474a-9f30-8384537e22ab",
    :exec-duration 390,
    :id "203f844b-df5e-41b8-b2b7-28887630762d",
    :kind "code",
    :output-log-lines {:stdout 3},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "216cfdf3-5285-4e88-b9f7-2c59fdb86d21"
   {:compute-ref #uuid "3d2035e7-2fb2-4318-a45e-97c752b92bbc",
    :exec-duration 431,
    :id "216cfdf3-5285-4e88-b9f7-2c59fdb86d21",
    :kind "code",
    :output-log-lines {:stdout 5},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "24164315-9939-4f8c-87da-ba4ea0706c2c"
   {:compute-ref #uuid "d905661d-e24d-45e9-95da-0e2d6c3e6796",
    :exec-duration 747,
    :id "24164315-9939-4f8c-87da-ba4ea0706c2c",
    :kind "code",
    :output-log-lines {:stdout 1},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "24514db5-6b38-4647-ae6e-6af8dd632f6a"
   {:compute-ref #uuid "a151210d-0226-43c2-a38b-dc667c209046",
    :exec-duration 363,
    :id "24514db5-6b38-4647-ae6e-6af8dd632f6a",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "2a4ef05e-0de6-4beb-b55f-a7dc1a91227d"
   {:compute-ref #uuid "046d5bda-c575-4afe-95c9-cc15280d4965",
    :exec-duration 12345,
    :id "2a4ef05e-0de6-4beb-b55f-a7dc1a91227d",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "2bdf86cd-0f2b-4fab-bd36-87261d6fe180"
   {:compute-ref #uuid "e3e66887-95a4-4a0d-9681-a18a53412194",
    :exec-duration 210,
    :id "2bdf86cd-0f2b-4fab-bd36-87261d6fe180",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "2f0ff0c8-78dc-47ba-9aca-8eda9a3d3372"
   {:compute-ref #uuid "5f602d9f-66a1-450b-81b1-9888484a3970",
    :exec-duration 650,
    :id "2f0ff0c8-78dc-47ba-9aca-8eda9a3d3372",
    :kind "code",
    :output-log-lines {:stdout 1},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "34d7ce65-9228-44c2-9a45-36d89c2f10fe"
   {:compute-ref #uuid "a5cbe5b0-e3a6-4ab6-bf89-c97f300c3fae",
    :exec-duration 381,
    :id "34d7ce65-9228-44c2-9a45-36d89c2f10fe",
    :kind "code",
    :output-log-lines {:stdout 10},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "363e8245-f397-4741-8cc7-29d312c72500"
   {:compute-ref #uuid "b82e28c2-5f1a-4b95-89ff-2a3c85e192d2",
    :exec-duration 1370,
    :id "363e8245-f397-4741-8cc7-29d312c72500",
    :kind "code",
    :output-log-lines {:stdout 2},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "3728d843-30c2-44d8-8c0a-d8e687bb3758"
   {:compute-ref #uuid "40f70d61-6743-42f3-8c54-e45437b99afe",
    :exec-duration 24990,
    :id "3728d843-30c2-44d8-8c0a-d8e687bb3758",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "3a3392d6-c871-4e32-bbed-374ada0ac49b"
   {:compute-ref #uuid "2460f4f1-b505-4613-8180-e21a4c25aca7",
    :exec-duration 7119,
    :id "3a3392d6-c871-4e32-bbed-374ada0ac49b",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "3a9e8c3d-46e3-4881-84d1-b15f729157c6"
   {:compute-ref #uuid "40ffffec-aa6b-4e30-bef3-79b069518884",
    :exec-duration 2973,
    :id "3a9e8c3d-46e3-4881-84d1-b15f729157c6",
    :kind "code",
    :output-log-lines {:stdout 5},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "3b0bfb28-8ae8-49e3-9075-bba71323c03d"
   {:compute-ref #uuid "5cffe330-9d8b-4b16-bf7c-2beb5daafd49",
    :exec-duration 2453,
    :id "3b0bfb28-8ae8-49e3-9075-bba71323c03d",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "3c18df72-b6e4-471d-9b1a-b12fb618c2b6"
   {:environment
    [:environment
     {:article/nextjournal.id
      #uuid "5b45e6f7-fe51-488a-b89b-1c8f74dfb387",
      :change/nextjournal.id
      #uuid "5d38a346-698b-47bc-a030-abfb012f0efd",
      :node/id "4e307b44-52c3-4f79-9f3d-23047859e2fa"}],
    :environment? true,
    :id "3c18df72-b6e4-471d-9b1a-b12fb618c2b6",
    :kind "runtime",
    :language "r",
    :name "R TOPLAS19",
    :resources {:machine-type "n1-standard-4"},
    :type :nextjournal,
    :docker/environment-image
    "docker.nextjournal.com/environment@sha256:0debe9e941d957540b440e9c118b7daaa9dbaab97c612342c0d041b394dab797",
    :runtime/mounts []},
   "3cb01fd1-9e27-41ae-ad98-9b08f12fb2d0"
   {:compute-ref #uuid "e04c1acf-adf2-4e17-8a4b-d711d3ef7bfc",
    :exec-duration 603,
    :id "3cb01fd1-9e27-41ae-ad98-9b08f12fb2d0",
    :kind "code",
    :output-log-lines {:stdout 1},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "3d0f85f3-894d-46b0-8d0d-b8cea873fd95"
   {:compute-ref #uuid "e45b2f3c-7d5c-46c9-83a7-bf50b9460c93",
    :exec-duration 23923,
    :id "3d0f85f3-894d-46b0-8d0d-b8cea873fd95",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "3f203c5b-2b45-42c2-ad5d-c75dbc8bde98"
   {:compute-ref #uuid "20cf7810-b53e-46e0-94bf-0217d571204c",
    :exec-duration 24725,
    :id "3f203c5b-2b45-42c2-ad5d-c75dbc8bde98",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "3f80cedb-d800-4133-a745-2b3ecf4d923a"
   {:compute-ref #uuid "fe4d5f22-29b6-401b-831d-1816fc8cd18a",
    :exec-duration 234,
    :id "3f80cedb-d800-4133-a745-2b3ecf4d923a",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "430c34d0-686b-4d21-a864-83e02ebb246f"
   {:compute-ref #uuid "574bdbe7-a52e-4ed7-a06d-b44afce09382",
    :exec-duration 193,
    :id "430c34d0-686b-4d21-a864-83e02ebb246f",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "4403efe0-1c1d-4faf-935b-c111c7644a3f"
   {:compute-ref #uuid "68151312-8402-4bc3-8219-b1bc987a6180",
    :exec-duration 203,
    :id "4403efe0-1c1d-4faf-935b-c111c7644a3f",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "46e51b59-b929-4539-b343-81af11447fe5"
   {:compute-ref #uuid "ed4a20aa-b7a1-4f26-b63c-f48bb523e97d",
    :exec-duration 319,
    :id "46e51b59-b929-4539-b343-81af11447fe5",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "47022958-8e10-454e-9ab6-8bd98f49c108"
   {:compute-ref #uuid "ceea904f-9c82-4a38-9956-772c6eae055d",
    :exec-duration 13392,
    :id "47022958-8e10-454e-9ab6-8bd98f49c108",
    :kind "code",
    :output-log-lines {:stdout 2},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "48742117-495b-40e9-8ed7-15f3f061b494"
   {:compute-ref #uuid "3eaa1dad-4db9-4e01-a6d4-32ecfd848203",
    :exec-duration 732,
    :id "48742117-495b-40e9-8ed7-15f3f061b494",
    :kind "code",
    :output-log-lines {:stdout 2},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "490f0c1f-fd45-4847-9a31-3cc789f2b9b8"
   {:compute-ref #uuid "482c34fd-6e93-4552-b3fd-e2b3c743e788",
    :exec-duration 2710,
    :id "490f0c1f-fd45-4847-9a31-3cc789f2b9b8",
    :kind "code",
    :output-log-lines {:stdout 1},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "497f38e6-82bf-44d4-90c5-4e25ba2fb8bd"
   {:compute-ref #uuid "f485c480-1f8a-4286-8f91-8ffdda534473",
    :exec-duration 5579,
    :id "497f38e6-82bf-44d4-90c5-4e25ba2fb8bd",
    :kind "code",
    :output-log-lines {:stdout 2},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "50218144-c124-476a-9c81-c78f4eed67fd"
   {:compute-ref #uuid "931bdd75-6af4-4c21-a7a6-d6dec1fa7779",
    :exec-duration 4511,
    :id "50218144-c124-476a-9c81-c78f4eed67fd",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "52c0b24f-3cd6-4ca4-ba41-2f000563abab"
   {:compute-ref #uuid "6bff6735-9394-4578-b197-af716128ca1b",
    :exec-duration 215,
    :id "52c0b24f-3cd6-4ca4-ba41-2f000563abab",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "557942a2-05a3-440d-92ad-fee3bacb0147"
   {:compute-ref #uuid "34d04798-0293-4aa6-9d89-80273ccaff48",
    :exec-duration 3028,
    :id "557942a2-05a3-440d-92ad-fee3bacb0147",
    :kind "code",
    :output-log-lines {:stdout 1},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "59edef53-67a8-4ad0-b212-6de2875c8ed4"
   {:compute-ref #uuid "52cf4eb4-3213-4f8b-9b64-37b4882c9e56",
    :exec-duration 246,
    :id "59edef53-67a8-4ad0-b212-6de2875c8ed4",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "5b1a8532-4eee-4521-940d-8cf5c3a122d7"
   {:compute-ref #uuid "bd5778c6-5d8f-4c96-b71f-bf24e41b03ab",
    :exec-duration 404,
    :id "5b1a8532-4eee-4521-940d-8cf5c3a122d7",
    :kind "code",
    :output-log-lines {:stdout 2},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "5c960584-dddb-4c51-bee0-00fd7721297b"
   {:compute-ref #uuid "cf80e240-57d0-418f-8e62-54a28cee2d4f",
    :exec-duration 326,
    :id "5c960584-dddb-4c51-bee0-00fd7721297b",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "5fc2098e-f67d-4885-9a29-f62eb73f9d6e"
   {:compute-ref #uuid "9f5d2814-416f-498e-8821-c638abe68a2d",
    :exec-duration 226894,
    :id "5fc2098e-f67d-4885-9a29-f62eb73f9d6e",
    :kind "code",
    :output-log-lines {:stdout 8733},
    :runtime [:runtime "3c18df72-b6e4-471d-9b1a-b12fb618c2b6"]},
   "60eb97d6-b9f8-48c3-8901-294edfa4893c"
   {:compute-ref #uuid "18c490b3-a3d7-4818-9245-b9dd72b2e29d",
    :exec-duration 449,
    :id "60eb97d6-b9f8-48c3-8901-294edfa4893c",
    :kind "code",
    :output-log-lines {:stdout 10},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "687707b5-7683-4dfd-aa96-9fbd33c2b6ac"
   {:compute-ref #uuid "0285bbdc-977b-4106-9734-06be9974bb47",
    :exec-duration 3160,
    :id "687707b5-7683-4dfd-aa96-9fbd33c2b6ac",
    :kind "code",
    :output-log-lines {:stdout 3},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "6920dcff-21c0-48c7-934d-46696eb94a3f"
   {:compute-ref #uuid "744506a1-b162-4e1e-bfa9-f5dccda61095",
    :exec-duration 337,
    :id "6920dcff-21c0-48c7-934d-46696eb94a3f",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "6d6c87a4-c7e9-4fe8-b8b0-d9739c28ca1a"
   {:compute-ref #uuid "6f8d949f-f081-4b83-bbbb-3c9d09e463fc",
    :exec-duration 676,
    :id "6d6c87a4-c7e9-4fe8-b8b0-d9739c28ca1a",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "6df945d4-528f-4437-80f6-21bd47f22da9"
   {:compute-ref #uuid "5c4a9406-beab-446a-87b9-20034bd50edb",
    :exec-duration 215,
    :id "6df945d4-528f-4437-80f6-21bd47f22da9",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "6f6fc6a0-fbe1-4875-8577-ba3546235487"
   {:compute-ref #uuid "644ef269-f486-47b7-bdc2-16ebb764f541",
    :exec-duration 3909,
    :id "6f6fc6a0-fbe1-4875-8577-ba3546235487",
    :kind "code",
    :output-log-lines {:stdout 2},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "70f27289-ae00-4623-89e4-0536127492f8"
   {:compute-ref #uuid "ea4c4ca5-9c2c-4c0d-80d2-6ce60e0a76f0",
    :exec-duration 404,
    :id "70f27289-ae00-4623-89e4-0536127492f8",
    :kind "code",
    :output-log-lines {:stdout 10},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "713cd3e2-0afc-474b-aeb0-4ae264260404"
   {:compute-ref #uuid "89709f98-802e-4f7f-937f-c7d735cb35d7",
    :exec-duration 461,
    :id "713cd3e2-0afc-474b-aeb0-4ae264260404",
    :kind "code",
    :output-log-lines {:stdout 1},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "7267c2aa-8f03-4789-83a2-912b91fe3bbd"
   {:compute-ref #uuid "419cbff9-b73f-4df1-b9e8-f3122e3d6a76",
    :exec-duration 316,
    :id "7267c2aa-8f03-4789-83a2-912b91fe3bbd",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "73aceb78-24b4-4c9d-bad7-01cf035d3d99"
   {:compute-ref #uuid "cd33da4f-b0e6-4769-b588-577d5514cd82",
    :exec-duration 310,
    :id "73aceb78-24b4-4c9d-bad7-01cf035d3d99",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "7408b823-3242-49f9-be62-901948ba1695"
   {:compute-ref #uuid "abdd476a-0218-495a-99a8-296e311892f8",
    :exec-duration 472,
    :id "7408b823-3242-49f9-be62-901948ba1695",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "75268cf0-cba3-450f-99b5-4cc82a0d0606"
   {:compute-ref #uuid "f673e942-ceb5-4c83-b351-abb2361d0f3a",
    :exec-duration 466,
    :id "75268cf0-cba3-450f-99b5-4cc82a0d0606",
    :kind "code",
    :output-log-lines {:stdout 4},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "75a31db9-fcc0-4575-9b24-b32676d773a7"
   {:compute-ref #uuid "3938c80b-920d-4372-a893-3b245d4cfde8",
    :exec-duration 1588,
    :id "75a31db9-fcc0-4575-9b24-b32676d773a7",
    :kind "code",
    :output-log-lines {:stdout 51},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "77f29bb3-dc3f-4f33-80c8-8b2e45ed5010"
   {:compute-ref #uuid "9ee3e643-27e3-4670-8cf3-13d1e4ae49c2",
    :exec-duration 54475,
    :id "77f29bb3-dc3f-4f33-80c8-8b2e45ed5010",
    :kind "code",
    :output-log-lines {:stdout 7},
    :runtime [:runtime "3c18df72-b6e4-471d-9b1a-b12fb618c2b6"]},
   "790589d6-69cd-4e2b-9d95-135a3eead0d5"
   {:compute-ref #uuid "53ff26b6-8a20-4c4e-8d91-5647767675c7",
    :exec-duration 194,
    :id "790589d6-69cd-4e2b-9d95-135a3eead0d5",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "7a4e89d3-7f36-43b2-8bca-9ea2fc4ccf6b"
   {:compute-ref #uuid "19bb24e7-5ca4-4ef6-9e03-55bc7c831e15",
    :exec-duration 4468,
    :id "7a4e89d3-7f36-43b2-8bca-9ea2fc4ccf6b",
    :kind "code",
    :output-log-lines {:stdout 11},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "7ab16acc-7b3f-4be8-8719-3bf561728667"
   {:compute-ref #uuid "91b587cf-b6f9-40bd-9c11-73f19110dfc4",
    :exec-duration 1151,
    :id "7ab16acc-7b3f-4be8-8719-3bf561728667",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "7bb48ab6-41c7-4963-b80f-c1542b2e2e3e"
   {:compute-ref #uuid "0693b752-5254-4fc7-9abc-80a34fa6b846",
    :exec-duration 678,
    :id "7bb48ab6-41c7-4963-b80f-c1542b2e2e3e",
    :kind "code",
    :output-log-lines {:stdout 1},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "7ddec0d4-fe94-41a6-a830-363f527d3b37"
   {:compute-ref #uuid "d10d9aba-a5e2-4ae8-a8a6-1768dca43301",
    :exec-duration 762,
    :id "7ddec0d4-fe94-41a6-a830-363f527d3b37",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "7edb7013-8ef8-4467-bc9f-f1167d0b87ed"
   {:compute-ref #uuid "6c452234-e3d2-4e49-8e7d-231178a33348",
    :exec-duration 501,
    :id "7edb7013-8ef8-4467-bc9f-f1167d0b87ed",
    :kind "code",
    :output-log-lines {:stdout 11},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "822223d7-d759-4c9b-bab4-36050bcf6421"
   {:compute-ref #uuid "00e123d7-5a92-4c42-a3b9-7a8bc05de00d",
    :exec-duration 114487,
    :id "822223d7-d759-4c9b-bab4-36050bcf6421",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "840221e6-8efb-4bcd-80e4-756d7ee1c704"
   {:compute-ref #uuid "194ace9b-ced2-4f53-8317-8d88d10735ba",
    :exec-duration 902,
    :id "840221e6-8efb-4bcd-80e4-756d7ee1c704",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "842e6de5-df0f-4fec-86f8-7c0b785f9ce9"
   {:compute-ref #uuid "8b932944-aedf-4385-9c87-866924c0af72",
    :exec-duration 1628960,
    :id "842e6de5-df0f-4fec-86f8-7c0b785f9ce9",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "847680ae-87d8-4cf5-a51a-1a89056ab4ca"
   {:compute-ref #uuid "a90abf69-8a50-436b-8515-947ffc8438a7",
    :exec-duration 22827,
    :id "847680ae-87d8-4cf5-a51a-1a89056ab4ca",
    :kind "code",
    :output-log-lines {:stdout 3},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "85463605-dd28-446d-b895-a86cedb1b27f"
   {:compute-ref #uuid "08f0ae41-f562-4b39-a4d1-7bcaf0f4e794",
    :exec-duration 50359,
    :id "85463605-dd28-446d-b895-a86cedb1b27f",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "8952f6fd-2180-42af-99df-263d993d405c"
   {:compute-ref #uuid "b6064d50-7e7a-4ec5-8852-a205fe671f9b",
    :exec-duration 10727,
    :id "8952f6fd-2180-42af-99df-263d993d405c",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "8960a280-4fbc-4803-80af-378d5c60a7fd"
   {:compute-ref #uuid "96a8e6e5-cff8-4739-98cb-dcb23324ea07",
    :exec-duration 550,
    :id "8960a280-4fbc-4803-80af-378d5c60a7fd",
    :kind "code",
    :output-log-lines {:stdout 1},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "8a8f1376-ee49-4d4f-906f-e14d53e6dd94"
   {:compute-ref #uuid "664de540-d929-432d-bebd-952d5d6eda32",
    :exec-duration 156,
    :id "8a8f1376-ee49-4d4f-906f-e14d53e6dd94",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "8b81d611-da90-4062-9a3c-8e29e3a300ee"
   {:compute-ref #uuid "7040c410-67d1-4d9f-95c8-a51549919eed",
    :exec-duration 244,
    :id "8b81d611-da90-4062-9a3c-8e29e3a300ee",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "8f178b20-135b-42ca-bec3-8f4b24051380"
   {:compute-ref #uuid "28a1e5d4-f4d0-4565-8f6d-4799c39c6fd9",
    :exec-duration 406,
    :id "8f178b20-135b-42ca-bec3-8f4b24051380",
    :kind "code",
    :output-log-lines {:stdout 5},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "93cd9dff-5a48-4751-8f8c-de31c3d21654"
   {:compute-ref #uuid "eba69766-0784-457e-82f4-9e28413c55ca",
    :exec-duration 653,
    :id "93cd9dff-5a48-4751-8f8c-de31c3d21654",
    :kind "code",
    :output-log-lines {:stdout 4},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "95a93fec-3d4a-4b1c-8419-ca54e94303be"
   {:compute-ref #uuid "df29fc81-f9dc-4189-8224-0ac9e4946f32",
    :exec-duration 1107,
    :id "95a93fec-3d4a-4b1c-8419-ca54e94303be",
    :kind "code",
    :output-log-lines {:stdout 7},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "97fbf0ee-a80d-45cf-a490-eac80bb14cf7"
   {:compute-ref #uuid "58f0d46b-1f7c-4a8f-9423-c302b0bd3351",
    :exec-duration 45040,
    :id "97fbf0ee-a80d-45cf-a490-eac80bb14cf7",
    :kind "code",
    :output-log-lines {:stdout 2},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "9a35a691-e515-4921-a093-caf7b5d16d8a"
   {:compute-ref #uuid "f459a1f7-0ef3-43aa-9863-0d4ba04be693",
    :exec-duration 440,
    :id "9a35a691-e515-4921-a093-caf7b5d16d8a",
    :kind "code",
    :output-log-lines {:stdout 10},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "9bfc7799-00ce-4fdb-bff3-ac820c06add1"
   {:compute-ref #uuid "44f2ef57-18c2-4f84-8e8d-ac6f1c5681f6",
    :exec-duration 484,
    :id "9bfc7799-00ce-4fdb-bff3-ac820c06add1",
    :kind "code",
    :output-log-lines {:stdout 32},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "9d946af6-e55e-41b8-acd7-841798dbf451"
   {:compute-ref #uuid "d39ff2ea-7fdf-490c-a9fb-fe97220285ab",
    :exec-duration 4168,
    :id "9d946af6-e55e-41b8-acd7-841798dbf451",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "9ee9ce77-439f-4001-bfac-b83b12779b29"
   {:compute-ref #uuid "78a57ef1-3b74-4508-b65e-8e23555bd5a2",
    :exec-duration 1331,
    :id "9ee9ce77-439f-4001-bfac-b83b12779b29",
    :kind "code",
    :output-log-lines {:stdout 2},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "9fa8a6ac-b8f2-4575-9498-b67e0b306d41"
   {:compute-ref #uuid "11168ceb-a44c-40dd-8552-ff094e0d8e2c",
    :exec-duration 19238,
    :id "9fa8a6ac-b8f2-4575-9498-b67e0b306d41",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "9ff6b827-7237-4f22-b5d4-0d7af9de2c53"
   {:compute-ref #uuid "19b699ab-9c2e-48fd-b497-05dbe403e27f",
    :exec-duration 447,
    :id "9ff6b827-7237-4f22-b5d4-0d7af9de2c53",
    :kind "code",
    :output-log-lines {:stdout 4},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "a65ca8e7-abca-4c4d-bef4-355324507c99"
   {:compute-ref #uuid "d166f5b1-2b4c-4922-a493-406d4a945658",
    :exec-duration 802,
    :id "a65ca8e7-abca-4c4d-bef4-355324507c99",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "a69c52bc-3752-4fb0-8996-682e07f4afab"
   {:compute-ref #uuid "9ed2983b-b1e8-4f16-b801-fe1a9c959fdb",
    :exec-duration 810,
    :id "a69c52bc-3752-4fb0-8996-682e07f4afab",
    :kind "code",
    :output-log-lines {:stdout 3},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "a6bce8cd-c677-41e6-a4a5-b08666417620"
   {:compute-ref #uuid "80bb6321-ebd1-476d-804c-0bbf0e164ff4",
    :exec-duration 1584,
    :id "a6bce8cd-c677-41e6-a4a5-b08666417620",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "a7ab23ce-4943-440a-bd06-9208cf6ffb53"
   {:compute-ref #uuid "59387cc4-4e79-47c8-b703-8d0e1f9448d9",
    :exec-duration 698,
    :id "a7ab23ce-4943-440a-bd06-9208cf6ffb53",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "a8179331-f861-4eb6-996c-8f6c20335fc6"
   {:compute-ref #uuid "9ac0a817-5e24-4a62-935a-45c81990b35b",
    :exec-duration 490,
    :id "a8179331-f861-4eb6-996c-8f6c20335fc6",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "aa6ebef6-36b2-4aa9-87fc-9bc4b5f2422d"
   {:compute-ref #uuid "36323bf9-b41a-40e1-8d28-6208eae0674b",
    :exec-duration 197,
    :id "aa6ebef6-36b2-4aa9-87fc-9bc4b5f2422d",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "aa71794f-8d95-40ac-898e-de10fe5f6388"
   {:compute-ref #uuid "05fd92b3-1527-470a-afdf-7ec2cae766f5",
    :exec-duration 199,
    :id "aa71794f-8d95-40ac-898e-de10fe5f6388",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "ab69e736-ea25-4208-af90-7cdc3de3befb"
   {:compute-ref #uuid "b61cf00a-d6b9-4939-8b79-4ed644454282",
    :exec-duration 405,
    :id "ab69e736-ea25-4208-af90-7cdc3de3befb",
    :kind "code",
    :output-log-lines {:stdout 11},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "b09318ab-6250-4c72-9f55-42429737228b"
   {:compute-ref #uuid "44779a77-dbb8-4bad-9bee-51e8bb1469dd",
    :exec-duration 234,
    :id "b09318ab-6250-4c72-9f55-42429737228b",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "b23d7d74-9998-45c0-b131-d5109e4178da"
   {:compute-ref #uuid "42e93cdb-e246-4048-bde9-2e81802f64fb",
    :exec-duration 175,
    :id "b23d7d74-9998-45c0-b131-d5109e4178da",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "b311cf2a-351f-4645-a50f-49c3d2461c1b"
   {:compute-ref #uuid "961b6e23-34ff-45d4-a4e4-3202a6fd82fa",
    :exec-duration 1569,
    :id "b311cf2a-351f-4645-a50f-49c3d2461c1b",
    :kind "code",
    :output-log-lines {:stdout 2},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "b44f3d02-19d1-4054-b249-4045eb3cc688"
   {:compute-ref #uuid "9e76c01c-4553-415f-9da8-b4653a5fc37c",
    :exec-duration 8808,
    :id "b44f3d02-19d1-4054-b249-4045eb3cc688",
    :kind "code",
    :output-log-lines {:stdout 3},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "b84a6c4b-3252-449e-a5f6-595dbe485166"
   {:compute-ref #uuid "f0cbb065-6a80-47b7-829f-e7eeafcdc88e",
    :exec-duration 36168,
    :id "b84a6c4b-3252-449e-a5f6-595dbe485166",
    :kind "code",
    :output-log-lines {:stdout 2},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"
   {:environment [:environment "3c18df72-b6e4-471d-9b1a-b12fb618c2b6"],
    :id "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d",
    :kind "runtime",
    :language "r",
    :name "Analysis",
    :resources {:machine-type "n1-standard-4"},
    :type :nextjournal,
    :runtime/mounts []},
   "be6f7148-73c9-40ae-bc72-6514b7cbe4b3"
   {:compute-ref #uuid "8b14eeec-4cfa-4ef3-bb96-63c0f4fef670",
    :exec-duration 323,
    :id "be6f7148-73c9-40ae-bc72-6514b7cbe4b3",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "beb7a112-e588-4864-b261-ca0eb39d913f"
   {:compute-ref #uuid "f78b7d79-2181-4aca-9d38-b64a336a4309",
    :exec-duration 747,
    :id "beb7a112-e588-4864-b261-ca0eb39d913f",
    :kind "code",
    :output-log-lines {:stdout 1},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "c29eb259-34a0-4c9a-9450-edc64cd55e10"
   {:compute-ref #uuid "96acdde4-95a2-4166-8993-d420d7ca8ad9",
    :exec-duration 238,
    :id "c29eb259-34a0-4c9a-9450-edc64cd55e10",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "c2ad7b4e-cd2b-47fb-85bb-f94265699a91"
   {:compute-ref #uuid "c106bb0f-70ab-4af7-a131-cb83bda44921",
    :exec-duration 71916,
    :id "c2ad7b4e-cd2b-47fb-85bb-f94265699a91",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "c384c48a-7e1a-4362-bb5e-9fc84d0472f2"
   {:compute-ref #uuid "75669d3f-76e9-4ccb-96dd-eea7254bd3be",
    :exec-duration 225,
    :id "c384c48a-7e1a-4362-bb5e-9fc84d0472f2",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "c3f62e2a-a4fa-462a-b030-38a12e49067e"
   {:compute-ref #uuid "5c050c94-a746-4135-867d-ccdc7fe7e086",
    :exec-duration 1530,
    :id "c3f62e2a-a4fa-462a-b030-38a12e49067e",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "c6924612-47e4-40cd-94e3-56c8f1bd6871"
   {:compute-ref #uuid "8ceec179-192f-46b1-bf8f-33d7387bdcfb",
    :exec-duration 182,
    :id "c6924612-47e4-40cd-94e3-56c8f1bd6871",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "c6dced07-3fab-4a37-a973-3aa1e75873dd"
   {:compute-ref #uuid "a12a11d7-88b7-4660-8cdd-da7b40508b3d",
    :exec-duration 360,
    :id "c6dced07-3fab-4a37-a973-3aa1e75873dd",
    :kind "code",
    :output-log-lines {:stdout 4},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "c9dfb915-c7e5-45f3-b066-7b651aff084f"
   {:compute-ref #uuid "55adfc7b-168c-4d0c-a0d7-3d2848fad720",
    :exec-duration 202,
    :id "c9dfb915-c7e5-45f3-b066-7b651aff084f",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "ca065d26-97b7-4aa2-947b-3017be5ba5fd"
   {:compute-ref #uuid "7ae7dcb7-e2be-4e60-8b25-0abe3faca026",
    :exec-duration 5090,
    :id "ca065d26-97b7-4aa2-947b-3017be5ba5fd",
    :kind "code",
    :output-log-lines {:stdout 2},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "cdd13d43-6b99-45b8-8c14-d50d1d6d481a"
   {:compute-ref #uuid "2c441d54-b313-4adb-ba6b-77ee82288582",
    :exec-duration 245,
    :id "cdd13d43-6b99-45b8-8c14-d50d1d6d481a",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "d24d3172-b9a3-49f0-80a3-04793815e689"
   {:compute-ref #uuid "bd848271-ed28-43e2-b527-101fcf4d55c8",
    :exec-duration 388,
    :id "d24d3172-b9a3-49f0-80a3-04793815e689",
    :kind "code",
    :output-log-lines {:stdout 7},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "d40b14e5-6271-447f-8880-028e9305ae69"
   {:compute-ref #uuid "e98a47b8-53ed-45d7-9c9e-fda05a46011f",
    :exec-duration 5868,
    :id "d40b14e5-6271-447f-8880-028e9305ae69",
    :kind "code",
    :output-log-lines {:stdout 7},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "d6032ea4-b6b7-456c-876b-8030e1d994d2"
   {:compute-ref #uuid "1c57a575-ec19-41de-b1ce-87bb51d06de5",
    :exec-duration 306,
    :id "d6032ea4-b6b7-456c-876b-8030e1d994d2",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "d703ad29-5f8c-4391-8965-3b8eff3d5221"
   {:compute-ref #uuid "cb3d5d2d-9895-45b0-b1d3-42ee8be93c59",
    :exec-duration 563,
    :id "d703ad29-5f8c-4391-8965-3b8eff3d5221",
    :kind "code",
    :output-log-lines {:stdout 2},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "d76ab2af-9134-4030-92c1-a08fd16109ad"
   {:compute-ref #uuid "8940ff2b-1fa4-4bf6-9323-c8a0dad253bd",
    :exec-duration 524,
    :id "d76ab2af-9134-4030-92c1-a08fd16109ad",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "daf873d3-bf4f-418e-8f75-2a3ff354b954"
   {:compute-ref #uuid "c2412115-2aa8-4760-ab08-b6043bfd6ff2",
    :exec-duration 391,
    :id "daf873d3-bf4f-418e-8f75-2a3ff354b954",
    :kind "code",
    :output-log-lines {:stdout 3},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "dd493f53-2ad1-4176-8b00-15ef8bbbf1ea"
   {:compute-ref #uuid "0a8f4075-adde-458b-96ea-a2933039e7d7",
    :exec-duration 8777,
    :id "dd493f53-2ad1-4176-8b00-15ef8bbbf1ea",
    :kind "code",
    :output-log-lines {:stdout 60},
    :runtime [:runtime "3c18df72-b6e4-471d-9b1a-b12fb618c2b6"]},
   "ddd0c143-9f39-4ae2-a050-fb4d2b3a29d4"
   {:compute-ref #uuid "8dcdf65f-ff7c-4e34-b8f7-94ed0f071128",
    :exec-duration 24433,
    :id "ddd0c143-9f39-4ae2-a050-fb4d2b3a29d4",
    :kind "code",
    :output-log-lines {},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "e00ac52a-7a95-4790-aaca-84de5f73d4bc"
   {:compute-ref #uuid "8ec59eb5-2b9a-476f-a2d2-9aaa8bc13805",
    :exec-duration 9690,
    :id "e00ac52a-7a95-4790-aaca-84de5f73d4bc",
    :kind "code",
    :output-log-lines {},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "e44c0fc6-aab8-4709-b803-a0f1a962d96e"
   {:compute-ref #uuid "fe915866-8b17-467c-922a-c6547c8fdfb9",
    :exec-duration 162,
    :id "e44c0fc6-aab8-4709-b803-a0f1a962d96e",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "e602305d-d33e-44c8-8253-53eb54113c29"
   {:compute-ref #uuid "954ffe48-0269-4af0-afea-71fbadb690de",
    :exec-duration 2964,
    :id "e602305d-d33e-44c8-8253-53eb54113c29",
    :kind "code",
    :output-log-lines {:stdout 1},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "e6e68ace-7464-4511-8c9d-57109c202ade"
   {:compute-ref #uuid "7ad08eb0-e071-4790-a8f7-38d45b1d062e",
    :exec-duration 936,
    :id "e6e68ace-7464-4511-8c9d-57109c202ade",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "e835d074-edc9-40d2-a974-4e598b2a7810"
   {:compute-ref #uuid "3268cad6-f002-4097-87fd-cc859472af42",
    :exec-duration 10641,
    :id "e835d074-edc9-40d2-a974-4e598b2a7810",
    :kind "code",
    :output-log-lines {:stdout 6},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "e9173cce-1fcf-4ea6-a12b-7572370ea761"
   {:compute-ref #uuid "e839b2b9-0d39-4994-8fc4-5d82f25973a7",
    :exec-duration 1457,
    :id "e9173cce-1fcf-4ea6-a12b-7572370ea761",
    :kind "code",
    :output-log-lines {:stdout 2},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "ea770f4f-b390-4d54-9422-8c3279b5690f"
   {:compute-ref #uuid "84bc7c77-00f0-4542-9976-031202cd71d3",
    :exec-duration 1923616,
    :id "ea770f4f-b390-4d54-9422-8c3279b5690f",
    :kind "code",
    :output-log-lines {:stdout 503},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "ea87e534-201f-4a02-9864-afa527d5e926"
   {:compute-ref #uuid "0595e4f3-ef5a-42df-a8d5-85dac1ebd96b",
    :exec-duration 296,
    :id "ea87e534-201f-4a02-9864-afa527d5e926",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "eadab99c-1fb7-4d9f-8fda-ea3c8f5fb16b"
   {:compute-ref #uuid "11dce791-7e12-417d-8995-fb3efaf54639",
    :exec-duration 176,
    :id "eadab99c-1fb7-4d9f-8fda-ea3c8f5fb16b",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "f018b3a1-e7a3-4d3a-8c34-8dcf1073f753"
   {:compute-ref #uuid "e544dd3a-306c-433e-bafb-ad9989f0fec7",
    :exec-duration 466,
    :id "f018b3a1-e7a3-4d3a-8c34-8dcf1073f753",
    :kind "code",
    :output-log-lines {:stdout 3},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "f35490a1-51cd-4200-96ab-1dc14d265847"
   {:compute-ref #uuid "ff6cc794-b660-460c-ace7-4ef93b945705",
    :exec-duration 242,
    :id "f35490a1-51cd-4200-96ab-1dc14d265847",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "f5f61678-e60d-49ab-9269-3dbf0761b667"
   {:compute-ref #uuid "5b455d24-b334-4ab3-a0db-920e5dbc3150",
    :exec-duration 348,
    :id "f5f61678-e60d-49ab-9269-3dbf0761b667",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "f768f3bc-7157-4bcb-90a5-70a2a06f4a9a"
   {:compute-ref #uuid "09f550f6-ede4-49ba-8750-d074a0e57b2a",
    :exec-duration 245,
    :id "f768f3bc-7157-4bcb-90a5-70a2a06f4a9a",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "f9524cd8-8dd1-4d91-8f68-69d366fc7d85"
   {:compute-ref #uuid "b3a7caee-32ed-4e92-bdb3-111d8927bbea",
    :exec-duration 127591,
    :id "f9524cd8-8dd1-4d91-8f68-69d366fc7d85",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "fb2319e2-1d3e-46da-86c7-2975bb5365b0"
   {:compute-ref #uuid "d288b4d8-9589-4733-8581-b74cab6eafd2",
    :exec-duration 527,
    :id "fb2319e2-1d3e-46da-86c7-2975bb5365b0",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]},
   "fe2273fd-38ef-443a-b042-e3ad1045e151"
   {:compute-ref #uuid "3cd8dc24-6ce7-4748-b8e8-9c148a451d4e",
    :exec-duration 180,
    :id "fe2273fd-38ef-443a-b042-e3ad1045e151",
    :kind "code",
    :output-log-lines {:stdout 0},
    :runtime [:runtime "bb92e0d4-ffd0-4148-9e94-84f9fa9e547d"]}},
  :nextjournal/id #uuid "02b925aa-1fa9-45c5-968f-5209c14b328c",
  :article/change
  {:nextjournal/id #uuid "5f3a4467-f925-4aea-9b4f-f20834b80ace"}}}

```
</details>