JUnit Reporter

[jamiebuilds]

I know, I know...

But there's a lot of infrastructure out there that depends on the JUnit XML format.

For example: Bitbucket Pipelines will display failed tests nicely if they are reported via JUnit/xUnit.

There's also stuff like CircleCI which uses the previous execution times in tests in order to automatically balance tests across machines.

I think it would be fine to build this as a separate tool, but the TAP output is not enough to do it. It would require an API with more information.

[novemberborn]

There was talk at JSConf.eu to come up with a new way of doing this. I reckon newline-delimited JSON.

Unfortunately I then went traveling and haven't had the time to follow up on this.

I did speak to somebody from CircleCI who expressed frustration that people often get the JUnit formatting wrong.

In short… I hear ya, and I'd like to have a non-TAP output format that has more information, but it won't be JUnit.

(I'm closing this issue for housekeeping purposes, but let's keep the conversation going.)

[sindresorhus]

@jamiebuilds What information is missing? We could embed arbitrary data in TAP if something is missing.

[jamiebuilds]

• <testsuite> - I would just use file paths
• <testsuite time="0.05"> - Time to run a test suite (file)
• <testcase time="0.01"> - Time to run a test case (test())
• <failure> Source location - /path/to/test.js:40 (t.assertion())

Overview here

[jamiebuilds]

@novemberborn Do you want to start a new TAP/JUnit-like spec along with people at CircleCI? I can also get people from Bitbucket Pipelines involved

[jamiebuilds]

Okay so I thought the idea was so good I kinda just went ahead and started it: https://github.com/jamiebuilds/zap

I'm happy to keep working on it, add others to it, reach out to other organizations, we can change it however people want, or even abandon it and join in on what others want to do.

[novemberborn]

Do you want to start a new TAP/JUnit-like spec along with people at CircleCI? I can also get people from Bitbucket Pipelines involved

Yes that's my long-term plan. Something that can provide a high-fidelity representation of test runner behavior, both for third-party formatters and tooling integration.

We set up https://github.com/testjswg but nobody's had the time yet to get it going.

[rishavsharan]

Hi folks.
I am from MS Azure DevOps and this issue is very interesting to us as well.
We would also love it if

• Ava had native support for junit output via a config variable
• Ava allowed us to set the test config via an environment variable like "AVA_OPTS" (as pytest does). This will allow us to get xml output from every CI/CD project using Ava without the user having to update their config manually.

I am not in favor of TAP (where the metadata is not standardized) or a completely new format (with low adoption). JUnit XML is an industry standard and is used by most of the CI/CD platforms.

Let me know what you guys think. If there is consensus on this, we would love to pitch in with contributions as required.

[jamiebuilds]

If AVA could output ZAP then you could easily write a tool to convert it into JUnit XML or TAP. But you couldn't do it the other way around (at least very well).

Every CI/CD service I've spoken to about this has been frustrated with the existing formats today.

If you want to contribute something better to the open source community I would suggest heading in a direction that will push the community forwards. Anyone could add JUnit support to AVA, I was already willing to do it, but it's going to take a CI service adopting ZAP (or some other new/better format) in order to push it across the industry.

Azure DevOps is in a very crowded ecosystem of very well established CI tools, it's going to have to do a lot of work to differentiate itself. ZAP will let you do something that no other CI service can do right now: Real time aggregate well-formatted results of dev tools.

[rishavsharan]

Hey Jamie
ZAP seems like a cool new format. Streaming real time results is something we are interested in as it enables the fast failing philosophy. I will be keeping an eye on ZAP and try to increase its visibility internally within my team.

But our focus is right now on serving the current developer need and for that being able to consume any-Unit XML is critical. Having Ava output the xml report will help the projects using Ava, build and release on DevOps Pipelines with minimum friction.

[jamiebuilds]

Well I think you already have your answer from this issue then

[1] #1878 (comment)

[rishavsharan]

Oh, I did see @novemberborn 's reply and decided to contribute to this conversation by bringing in our perspective as a CI/CD platform. 👍

[novemberborn]

Hi @rishavsharan, thanks for chiming in. I can see the usefulness, however we won't support any output format that has lower fidelity than we (want to) achieve in our own logger.

We're open to supporting a NDJSON-based format that captures the richness of events we can (or want to be able to) output. I don't particularly care how much of a standard it is, or who standardizes it. I'd then like to see other tools to translate that output into TAP, JUnit, etc.

[bitliner]

I've found this https://www.npmjs.com/package/tap-junit

npm i -D tap-junit

./node_modules/.bin/ava -t | ./node_modules/.bin/tap-junit > test.xml

In my case it is working fine for now (using it for gitlab test reporter)