v0.12.0

General

The focus of this release was usability improvements for test runners.

Test Scheduling Priorities

We added a new priority field to the job struct to allow the client to indicate
higher-priority jobs. The tests runners now use this field to elevate the
priorities of new and previously-failed tests. This way, users get immediate
feedback about whether or not their changes fixed their tests.

New Configuration Values

We added a stop-after configuration value to all of the test runners. The
test runner will stop executing after this many failures. This pairs well with
the new scheduling priorities. Running cargo maelstrom --stop-after=1 will
give immediate feedback on whether your new tests failed, or whether your
changes intended to fix a failing test worked.

Passing Extra Arguments to Tests

We added a number of configuration values to allow the passing of arbitrary
arguments to test binaries. In the case of Pytest, we also added configuration
values to enable passing arbitrary arguments to the collect phase, the
test-running phase, or both.

The extra arguments can also be passed after -- on the command line, like
this: maelstrom-go-test -- -test.parallel 12.

More maelstrom-go-test Features

We added the vet, short, and fullpath pass-through configuration values
to maelstrom-go-test. We also added maelstrom-go-test --list-packages.

Better Default Metadata

We updated the default metadata for the test runners to be more useful. Each
test runner now has its own, specific, default metadata.

cargo maelstrom now passes RUST_BACKTRACE and RUST_LIB_BACKTRACE to the
test binaries.

maelstrom-pytest now has much better defaults, as is documented
here.

UI Improvements

The test-runner UI has been improved in a number of ways to provide more
information about the state of tests. See below for more information.

Shared-Library Dependencies

We added a new layer type called shared-library-dependencies, which includes
the closure of shared libraries required to run a list of binaries. This makes
it easier to include external binaries and their dependencies in test
containers.

Added

• maelstrom-go-test now edits out some lines from test failures that come
  from the go test test fixture code.
  [369]
• maelstrom-go-test now shows tests which call t.Skip as status IGNORED.
  [367]
• vet config value added to maelstrom-go-test.
  [356]
• short config value added to maelstrom-go-test.
  [354]
• fullpath config value added to maelstrom-go-test.
  [351]
• --list-packages command-line option to maelstrom-to-test.
  [318]
• extra-test-binary-args config value added to maelstrom-go-test.
  [363]
• extra-test-binary-args config value added to cargo-maelstrom.
  [370]
• extra-pytest-args config value added to maelstrom-pytest.
  [371]
  [372]
• extra-pytest-collect-args config value added to maelstrom-pytest.
  [372]
• extra-pytest-test-args config value added to maelstrom-pytest.
  [372]
• shared-library-dependencies layer type added.
  [266]
• stop-after config value added.
  [114]
• Scheduling priorities. Tests are run in LRT (longest remaining time) order
  within their priority band. Test with higher priority are always run before
  tests with lower priorities.
• Tracking of whether or not tests failed the last time they were run. If
  --repeat was specified, a test is considered to have failed if any instance
  failed.
• Functionality where new tests and tests that failed previously are run before
  other tests. This function pairs wells with the new stop-after config value.
  [380]
• Fancy UI lists failed tests in a dialog before it has finished running tests.
  [391]
• Simple and Fancy UI indicate the number of failed tests in their respective progress bars.
  [391]

Changed

• ui test-runner config option value now matches the same casing as the CLI option
  (kebab-case).
• UI now has improved tracking of the state of tests. The fancy UI in particular will now only show
  a test as running once it has actually started executing.
• quiet config option removed from test runners, replaced with new "quiet" UI.
  [374]
• Demoted warning to debug message concerning un-built test binaries in cargo-maelstrom.
  [359]
• Forward RUST_BACKTRACE and RUST_LIB_BACKTRACE by default in cargo-maelstrom.
  [386]
• maelstrom-pytest default test metadata has been updated to be better and contain examples
  [324]
  [326]
• A mount_point of "/" is now disallowed when parsing devices in test metadata files. This didn't
  produce a job that worked previously, this now just fails earlier.
  [344]
• A job with local networking and a "sys" mount now errors with a nicer message before even running.
  [389]

Fixed

• Fixed issue with test-runner fancy UI where it would crash if the summary was too large.
• Fixed issue with test-runners including extra trailing line in test output.
• Fixed issue where the helpful extra comments in the file written by test runners with --init
  weren't being added.

Removed

• Removed support for the deprecated maelstrom-test.toml file.
  [333]