README.Rmd

---
output: dynbenchmark::github_markdown_nested
editor_options: 
  chunk_output_type: console
---

```{r setup, include=FALSE}
library(tidyverse)
library(dynbenchmark)
dynbenchmark:::install_fontawesome()
extrafont::loadfonts()

output <- read_rds(result_file("output.rds", "04-method_testing"))

tab <- 
  output %>%
  select(method_id, dataset_id, method_status) %>%
  mutate(dataset_id = gsub("specific_example/.*", "specific_example", dataset_id)) %>% 
  left_join(dynbenchmark::method_statuses, by = "method_status")
```

# Quick testing of methods using small datasets

We test the methods on:

- A specific example dataset specifically created for this method. The code to generate this dataset is included in the repository of each method, for example: [https://github.com/dynverse/ti_scorpius/blob/master/example.R](https://github.com/dynverse/ti_scorpius/blob/master/example.R). This dataset is also used in the continuous integration of the method, for example: [https://travis-ci.org/dynverse/ti_paga](https://travis-ci.org/dynverse/ti_paga).
- Two real and two synthetic datasets, which all clearly contain a trajectory on a dimensionality reduction.

Only methods which do not fail on at least one dataset will be used in subsequent evaluations.

```{r}
g <- ggplot(tab, aes(dataset_id, fct_rev(method_id))) + 
  geom_tile(aes(fill = method_status)) + 
  geom_text(aes(label = fa, colour = method_status), family = "Font Awesome 5 Free", size = 3) +
  scale_fill_manual(values = dynbenchmark::method_statuses %>% select(method_status, colour) %>% deframe) +
  scale_colour_manual(values = dynbenchmark::method_statuses %>% select(method_status, icon_colour) %>% deframe) +
  coord_equal(expand = FALSE) +
  theme_classic() +
  labs(x = NULL, y = NULL, fill = "Method status", colour = "Method status") +
  theme(axis.text.x = element_text(angle = -90, hjust = 0))

add_fig(
  g,
  "method_testing_figure",
  "Execution status of each of the TI methods on a small set of real and toy datasets.",
  width = 6,
  height = 20
)

```