assert-* command should not result in test error (but failure)

[rbri]

% migrated from trac

Idea: make all assert-* commands failure only commands

Preambel
Wetator distinguishes between error and failure:
errors immediately stop test execution (with some exception, e.g. exec-java commands).
failures are noted, but test execution continues.
errors mark considerable problems that cancel the validity of subsequent test steps (e.g. commands controlling the test flow like open-url, click-on, set, ...). failures mark problems without effect on subsequent test steps.

Status Quo
Failing assert-content and assert-title commands always result in failures (never errors)
while other failing assert-* commands (e.g. assert-set or assert-selected) sometimes result in failures, but may also result in errors. They result in errors, when the aimed control cannot be found at all.

Target State(?)
My concern comes from usability (and comprehension) point of view and I guess that those cases should be failures instead of errors, so that test execution can continue; although I understand the initial (technical) idea of the current behavior.

Example:
Consider a test, testing 20 assert-set commands, to test correct rendering and contents of 20 input fields on a page of your application. The 2nd, 4th and 19th input field are not rendered.

Current behavior: Test breaks with assert-set error for the 2nd input field. You adjust your code and run again. Test breaks with assert-set error for the 4th input field. You adjust your code and run again. Test breaks with assert-set error for the 19th input field. You adjust your code and run again. Test successful.

Desired behavior: Test breaks with 3 failures (one run only!). You adjust your code for all three inputs and run again. Test successful.

What we'd lose: The page we are testing may be completely invalid (even the wrong page) what we only get to know with the expressiveness of an error if at least one controlling command would follow the failing assert- command. But: we get a failure anyway and need to have a look at our test. So this would be an execution time issue only while we would gain a more complete test execution for all those cases where our page is valid.