Report test results using Open Test Reporting XML format

[marcphilipp]

TL;DR The JUnit team has defined a new language-agnostic test reporting format and implemented a CLI tool for validation, conversion, and HTML report generation. We're reaching out to well-known testing frameworks and reporting tools to ask for feedback and, ultimately, adoption, if you think this format provides value to your users.

Motivation and Context

You've probably come across the "JUnit XML" format for test reporting. This format did not originate from the JUnit project but was initially introduced by the Ant build tool and then adopted by other Java build tools like Maven and Gradle. Many build servers know how to parse the XML-based format, and even non-Java tools often support it. However, it’s based on the concept of test classes and methods, so using it for frameworks and tools where those elements are not present is awkward at best. Moreover, it does not support nested structures beyond a simple parent-child relationship. Finally, it is not extensible: no additional attributes can be added without the risk of breaking existing tools.

For those reasons, many testing frameworks (for example, TestNG and Spock in the Java ecosystem) have defined their own reporting formats. This has given them the flexibility they need, but the number of tools that can parse, display, or transform their custom formats is very limited.

To overcome these limitations, the JUnit team is defining a new format for test reporting. Its goal is to be platform-agnostic so that as many testing frameworks as possible can benefit from it. Moreover, it is designed to be extensible so new data can be added as needed, without breaking consumers. However, all well-known attributes are properly defined so it remains consumable by downstream reporting tools.

Of course, it will take a while for downstream tools to support the new format. However, as the number of testing frameworks that have adopted it increases, the more likely downstream tools are to do so as well.

Overview

The new format is based on XML because it provides more expressive ways to define schemas. Moreover, XML has typed extensions built-in via the use of multiple schemas. If a testing framework provides a listener mechanism, it should be possible to write an Open Test Reporting XML file from an extension.

Benefits

• Easy to write event-based XML format (with optional Java API)
  • Supports infrastructure information, Git metadata, tags, data/file attachments
  • Custom data can be added by defining an extension schema
• Language-agnostic
  • No Java-specific elements in the core schema
  • Not tied to the concept of test classes and methods
  • Full support for nested structures
• Extensible HTML report generator

Next Steps

The JUnit team would be happy to get your feedback on this initiative. We can discuss here or you're welcome to start a thread or open an issue in the Open Test Reporting repo. Should you consider adopting the new format, we'd be happy to provide guidance but we won't have the resources to actually contribute an implementation.

[sebastianbergmann]

Thank you for letting us know that the Open Test Reporting format is now ready. Last time we talked about this you were just starting with the specification.

We will look into adding support for this, but I cannot make a promise as to if/when we will do this. The event-based extension system introduced in PHPUnit should allow for this to be implemented outside of PHPUnit's core in an extension for its test runner.

I have not yet looked at your specification, but I fear that there may be information specific to PHP and PHPUnit that we cannot represent with it. For instance, with PHPUnit a test may have one outcome (success, failure, error, skipped, incomplete) as well as >=0 issues (triggered deprecation in tested code, triggered PHP deprecation, triggered PHPUnit deprecation, was considered risky, ...).

I had planned to finally implement #4321 (together with @theseer) next year. Part of that plan now includes looking into your specification. Would be great if we did not need to come up with our own XML format after all.

[marcphilipp]

I have not yet looked at your specification, but I fear that there may be information specific to PHP and PHPUnti that we cannot represent with it. For instance, with PHPUnit a test may have one outcome (success, failure, error, skipped, incomplete) as well as >=0 issues (triggered deprecation in tested code, triggered PHP deprecation, triggered PHPUnit deprecation, was considered risky, ...).

The format is extensible. Even JUnit has its own schema to add JUnit-specific metadata. Moreover, the HTML report generation provides an SPI (for Java) to include such custom information.

That being said, the issue data sounds like it might even be a good addition to the core schema or a centrally-maintained extension schema. I'd be happy to discuss when you've had time to take a closer look.

[sebastianbergmann]

Here is my current plan for implementing Open Test Reporting support using its event-based XML format:

• Implement --log-otr <file> CLI option (already implemented in the open-test-reporting branch)
• Implement <otr outputFile="logfile.xml"/> child child element for the XML configuration file's <logging> element (already implemented in the open-test-reporting branch)
• Implement writing a minimal logfile that is compliant with core-0.2.0.xsd and events-0.2.0.xsd (proof-of-concept implementation in the open-test-reporting branch)

Once the above is stable and proven to work, we may include Open Test Reporting support in a stable release of PHPUnit.

---

• Define php.xsd for PHP runtime information
• Include PHP runtime information in XML logfile written by PHPUnit (depends on php.xsd, see above)
• Define phpunit.xsd for information specific to PHPUnit
• Include information specific to PHPUnit in XML logfile written by PHPUnit (depends on phpunit.xsd, see above)

Once the above is stable and proven to work, we will include Open Test Reporting support in a stable release of PHPUnit, potentially expanding the minimum-viable version that we may have shipped in a previous stable release (see above).

---

Once PHPUnit supports Open Test Reporting as outlined above, we will deprecate its functionality to report test results in JUnit XML format in the next major version. The next major version of PHPUnit after that will remove the functionality to report test results in JUnit XML format. By that time, common consumers of JUnit XML should have gained native support for consuming Open Test Reporting XML. Alternatively, I expect tooling for converting Open Test Reporting XML to JUnit XML to exist.