Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Many testing situations call for more fine-grained results than a simple binary success/fail -- for instance, there may be optional capabilities a participant may choose to implement or not, and knowing whether each participant has been checked out for each optional capability may be useful. To address this need, this PR introduces the concept of "badges" which are explained in the uss_qualifier README.md changes.
From a technical perspective, available badges are defined by test suites, and those badges are granted to participants at the conclusion of running the test suite in which they are defined. The new
participant_badges
field in TestSuiteDefinition can be seen in suites/definitions.py and then the core badge data structures can be found in reports/badge_definitions.py. Badges are granted at the conclusion of a test suite run in suites/suite.py which calls out to badge grant logic in reports/badges.py.To support more precise identification of collections of requirements, RequirementsCollection is created in requirements/definitions.py, and the RequirementsID and RequirementsSetID data structures are moved into that module to support inclusion in other data structures while avoiding circular dependencies. The resolution of a RequirementsCollection into a set of requirements is added to the end of requirements/documentation.py.
RequirementSetIDs get an expanded capability to refer to a particular section of a Markdown document to more flexibly identify requirement collections.
All types of reports receive additional check examination abilities (finding all passed checks, failed checks, relating to a particular participant) to support badge grant logic, and some improvement is made to the way the pseudo-polymorphic TestSuiteActionReport accesses its different contents possibilities.
A bug in jsonpath-ng lead to the discovery that that package has not been updated in years, so it is replaced by a fork that fixes that bug and has been updated much more recently (bc-jsonpath-ng).
Example badges are added to a few test suites to verify functionality.
In a follow-up PR, I hope to add the capability to report details about why a particular badge was not granted to a particular participant to the output report as I suspect investigation of the reasons a badge wasn't granted will be a core user journey for participants under test.