Skip to content

Commit

Permalink
Add missing documentation for ReBenchDB
Browse files Browse the repository at this point in the history
Signed-off-by: Stefan Marr <[email protected]>
  • Loading branch information
smarr committed Sep 4, 2024
1 parent 48374b9 commit 0b994f7
Show file tree
Hide file tree
Showing 3 changed files with 37 additions and 16 deletions.
27 changes: 23 additions & 4 deletions docs/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ default_data_file: my-experiment.data
**artifact_review:**

Enable [artifact evaluation](https://www.artifact-eval.org/) mode.
This mode does report less warnings and errors, but does not change
This mode reports fewer warnings and errors, but does not change
how benchmarking is done. Experience shows that reviewers may misunderstand
possibly chatty warnings and misinterpret them as a signs for an artifact of
insufficient quality.
Expand Down Expand Up @@ -419,17 +419,19 @@ runs:
retries_after_failure: 3
```

---

## Reporting

Currently, [Codespeed] is the only supported system for continuous
performance monitoring. It is configured with the `reporting` key.
[Codespeed] and [ReBenchDB] are the currently supported system for continuous
performance monitoring. They are configured with the `reporting` key.

**codespeed:**

Send results to Codespeed for continuous performance tracking.
The settings define the project that is configured in Codespeed, and the
URL to which the results will be reported. Codespeed requires more information,
but since these details depend on the environment, other settings are passed via
but since these details depend on the environment, additional details are passed via
the [command line](usage.md#continuous-performance-tracking).

Example:
Expand All @@ -443,6 +445,23 @@ reporting:

---

**rebenchdb:**

Send results to a ReBenchDB server for continuous performance tracking.
The use of ReBenchDB will also require additional details that are passed via
the [command line](usage.md#continuous-performance-tracking).

```yaml
reporting:
rebenchdb:
db_url: https://example.org/rebenchdb
repo_url: https://example.org/project.git
project_name: Some Project that Wants Good Performance
record_all: true
```

---

## Benchmark Suites

Benchmark suites are named collections of benchmarks and settings that apply to
Expand Down
23 changes: 14 additions & 9 deletions docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,33 +156,38 @@ options:

#### Continuous Performance Tracking

ReBench supports [Codespeed][1] as platform for continuous performance
tracking. To report data to a Codespeed setup, the [configuration](config.md#codespeed)
ReBench supports [Codespeed][1] and [ReBenchDB][2] as platforms for continuous performance
tracking. To report data to such a setup, the [configuration](config.md#reporting)
needs to have the corresponding details.

And, Codespeed needs details on the concrete execution:
Additionally, Codespeed and ReBenchDB need details on the concrete execution:

```text
--commit-id=COMMIT_ID MANDATORY: when Codespeed reporting is used, the
--commit-id COMMIT_ID MANDATORY: when Codespeed reporting is used, the
commit-id has to be specified.
--environment=ENVIRONMENT MANDATORY: name the machine on which the results are
--environment ENVIRONMENT MANDATORY: name the machine on which the results are
obtained.
--branch=BRANCH The branch for which the results have to be recorded,
--branch BRANCH The branch for which the results have to be recorded,
i.e., to which the commit belongs. Default: HEAD
--executable=EXECUTABLE The executable name given to Codespeed. Default: The
--executable EXECUTABLE The executable name given to Codespeed. Default: The
name used for the executor.
--project=PROJECT The project name given to Codespeed. Default: Value
--project PROJECT The project name given to Codespeed. Default: Value
given in the config file.
-I, --disable-inc-report Creates a final report at the end instead of reporting
incrementally.
-R, --disable-data-reporting Override the configuration and disable any reporting
to Codespeed and ReBenchDB.
--git-repo GIT_REPO Path to the git repository that contains the source of this experiment.
This is useful when the experiment is run from a different path
without the `.git` folder.
```

[1]: https://github.com/tobami/codespeed/
[1]: https://github.com/tobami/codespeed
[2]: https://github.com/smarr/ReBenchDB
3 changes: 0 additions & 3 deletions rebench/rebench-schema.yml
Original file line number Diff line number Diff line change
Expand Up @@ -91,9 +91,6 @@ schema;reporting_type:
project_name:
type: str
desc: Project name to be used by ReBenchDB.
repo_base_branch:
type: str
desc: The name of the base branch for comparisons.
record_all:
type: bool
desc: All experiments should be stored in the ReBenchDB
Expand Down

0 comments on commit 0b994f7

Please sign in to comment.