NEW: add code coverage to CI jobs running Editor and Playmode tests on editor platforms

[stefanunity]

Description

• Win, Mac & Linux editor jobs now run code coverage for the package integration tests & project tests (Assets/Tests).
• coverage reports are part of the artifacts being generated & zipped

Testing status & QA

• CI is green
• reports are generated and appear in artifacts

Overall Product Risks

No product risk, test jobs take slightly more time.

• Complexity: Low
• Halo Effect: Low

Checklist

Before review:

• Changelog entry added.
  • Explains the change in Changed, Fixed, Added sections.
  • For API change contains an example snippet and/or migration example.
  • JIRA ticket linked, example (case %%). If it is a private issue, just add the case ID without a link.
  • Jira port for the next release set as "Resolved".
• Tests added/changed, if applicable.
  • Functional tests Area_CanDoX, Area_CanDoX_EvenIfYIsTheCase, Area_WhenIDoX_AndYHappens_ThisIsTheResult.
  • Performance tests.
  • Integration tests.
• Docs for new/changed API's.
  • Xmldoc cross references are set correctly.
  • Added explanation how the API works.
  • Usage code examples added.
  • The manual is updated, if needed.

During merge:

• Commit message for squash-merge is prefixed with one of the list:
  • NEW: ___.
  • FIX: ___.
  • DOCS: ___.
  • CHANGE: ___.
  • RELEASE: 1.1.0-preview.3.

After merge:

• Create forward/backward port if needed. If you are blocked from creating a forward port now please add a task to ISX-1444.

[unity-cla-assistant]

All committers have signed the CLA.

[ekcoh]

@stefanunity I unzipped "artefacts" in Yamato on this PR but could find any coverage report there, where may I find it?

[ekcoh]

@stefanunity Maybe its only on specific jobs? You mention integration tests (not many tests) and project tests?

[stefanunity]

@stefanunity Maybe its only on specific jobs? You mention integration tests (not many tests) and project tests?

See DM with artifact links.

[lyndon-unity]

Great to get this added. Thank you.

[ekcoh]

Thanks for helping find the coverage report jobs @stefanunity. Fist of all, great to get this added. However, when reviewing the generated report, e.g. looking at HTML report https://yamato-artifactviewer.prd.cds.internal.unity3d.com/a62e0ec1-0e59-4934-b3f4-b15136f269d1%2FUTR_Output.zip%2Fupm-ci~%2Ftest-results%2FCodeCoverage%2FPackage%2FReport/index.html (Win job), I notice code coverage is extremely low. Picking a certain random class, e.g. GamepadState, we can see that code coverage is 0% in the report, another example I checked was PressedInteraction which also report 0% coverage. Putting a breakpoint inside these classes and running the test suite in editor locally hits those method from the test suite so something is wrong. (Play mode tests)

I suspect, maybe a merge operation of cobertura reports isn't done to get a merged report before calling report tool? And the current result we see is likely only editor tests? Since there are multiple test runs their results should be merged. It might be that coverage is only indicating a partial runs results?

[ekcoh]

Looking at the artefacts there seem to be 4 xml reports that need to be merged before generating report with ReportTool unless it support multiple sources on CLI interface.

[stefanunity]

Looking at the artefacts there seem to be 4 xml reports that need to be merged before generating report with ReportTool unless it support multiple sources on CLI interface.

Resolved as per convo in sync meeting.