The Easiest Way to Generate and Publish .NET Code Coverage in Azure DevOps | josh-ops

[utterances-bot]

The Easiest Way to Generate and Publish .NET Code Coverage in Azure DevOps | josh-ops

Publishing Code Coverage and making it look pretty in Azure DevOps is way harder than it should be

https://josh-ops.com/posts/azure-devops-code-coverage/

[msawayda]

Just a heads up. If using the script snippet to add reportgenerator it will publish the reports to
$(Build.SourcesDirectory)/CodeCoverage

Then when you use the PublishCodeCoverageResults@1 snippet below that it will try to reference it in
$(Build.SourcesDirectory)/CoverageResults.

Those both have to be the same directory for it to work.

[joshjohanning]

Nice tip @msawayda, thank you!

[VincentOspazi]

Hi Josh,

When multiple unit test projects are present, ;you can also merge the generated reports by adding the following parameter in the test run of the last project to test.

--merge-with $(Agent.TempDirectory)/**/coverage.cobertura.xml

And then of course afterwards use the PublishCodeCoverageResults task

[joshjohanning]

you can also merge the generated reports by adding the following parameter in the test run of the last project to test.

--merge-with $(Agent.TempDirectory)/**/coverage.cobertura.xml

@VincentOspazi Do you have a broader example you can share? I couldn't find much about --merge-with only /p:MergeWith=.

[VincentOspazi]

Hi @joshjohanning, for ex ample:

you can find it in the doc of coverlet : https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/GlobalTool.md

- task: DotNetCoreCLI@2
  displayName: 'Run Unit Tests for Shared module'
  inputs:
    command: test
    projects: $(sharedLocation)Tests/*.Tests.csproj
    arguments: '--configuration $(buildConfiguration) --no-restore --filter Category=UnitTest --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'

- task: DotNetCoreCLI@2
  displayName: 'Run Unit Tests for Core module'
  inputs:
    command: test
    projects: '$(coreLocation)/Tests/*.Tests.csproj'
    arguments: '--configuration $(buildConfiguration) --no-restore --filter Category=UnitTest --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura,opencover --merge-with $(Agent.TempDirectory)/**/coverage.cobertura.xml'

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage'
  inputs:  
    codeCoverageTool: Cobertura
    summaryFileLocation: '$(Agent.TempDirectory)/**/coverage.cobertura.xml'

the last test command we output to both cobertura (for Azure DevOps) and opencover (for SonarQube to grab).

Hope this helps.

[james1301]

Thanks Josh I think I have got this up and working now. And the code coverage tab is good to have. I am struggling to find much detail on examples or anything but how to actually post something on the PR to this, even like just the Code Coverage % or something?

Microsoft examples talk about only working with their standard coverage format which isn't too helpful.

[joshjohanning]

@james1301 You could perhaps use an extension or the API to post at least the code coverage % as a comment in the Pull Request?

Maybe one of these would help:
https://stackoverflow.com/questions/60048492/how-to-create-a-comment-in-azure-devops-pr-in-case-of-build-failure
https://marketplace.visualstudio.com/items?itemName=tylermurry.pr-auto-comment
https://marketplace.visualstudio.com/items?itemName=CSE-DevOps.create-pr-comment-task

Use a condition to only run the task when triggered via PR.

You would then just have to extract the code coverage from the coverage.cobertura.xml file to write as a comment; ie:

<coverage line-rate="0.21974657217686028" 

Something like this might help:
https://unix.stackexchange.com/questions/529670/extract-an-attribute-value-from-xml

Example of coverage xml:
https://gist.github.com/apetro/fcfffb8c4cdab2c1061d

[rosdi]

This got my code coverage working..., thanks a lot!

[rosdi]

Is there a way to fail the build if code coverage is below a certain percentage? Say below 70%?

[joshjohanning]

This got my code coverage working..., thanks a lot!

Great!

Is there a way to fail the build if code coverage is below a certain percentage? Say below 70%?

@rosdi Yes! There are several extensions on the marketplace that bring in tasks for this, but my favorite are Colin's ALM Corner Custom Build Tasks, which has a Coverage Gate task. See more information on it here.

[rosdi]

Interesting..., didn't know it is called Coverage Gate. Will surely check these out.

[joshjohanning]

Great @rosdi. It would certainly be worth an addition to this post to call this task out 😄.

[terryaney]

FYI: I don't know if you know where to find source code for the tasks or not (based on your response about only finding information on p:MergeWith, but for whatever reason, if I do NOT use your command line arguements for dotnet test but use the following:

--configuration Release --no-build --logger trx;LogFileName=TestResults.trx /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:ExcludeByFile=**/LoggerMessage.g.cs /p:CoverletOutput=TestResults/coverage.cobertura.xml

Only one version of the *.xml and *.trx are generated in the /TestResults folder relative to the test csproj file.

[joshjohanning]

@terryaney thanks for sharing! In your example, does that consolidate/merge the results of multiple test projects into one file?

I think in @VincentOspazi's example, they were merging in multiple coverlet/cobertura results into one with the --merge-with argument which is from coverlet itself it seems.

[terryaney]

No, but as you stated in your post, the reportgenerator can take multiple file inputs in its -reports flag. So I assume those will merge into a single report. I have to finish my CICD build pipelines of a project that has multiple tests to confirm this.

[joshjohanning]

No, but as you stated in your post, the reportgenerator can take multiple file inputs in its -reports flag. So I assume those will merge into a single report. I have to finish my CICD build pipelines of a project that has multiple tests to confirm this.

@terryaney ahhh, I see now. Yes, most definitely you could use the reportgenerator task/CLI to merge them all after the fact 😄 . That's probably the easiest way.

Based on coverlet's docs, it seems like your method is using MSBuild integration as opposed to the way I was referring to it in my post (with the VS Test Platform). If you only cared about the results posting to Azure DevOps and not the --logger trx, I'd presume you could drop that part of your command perhaps.

---

And I see now after second a second look in @VincentOspazi's example, they are running dotnet test twice, first generating a cobertura test results file, then they are running dotnet test against the second project with the --merge-with to add the results of the second test to the first test file.

[joshjohanning]

Is there a way to fail the build if code coverage is below a certain percentage? Say below 70%?

@rosdi For posterity, posting this here. I found this in the docs while looking up for the other question - you can fail the build natively if you use the MSBuild integration method.

You can use:

dotnet test /p:CollectCoverage=true /p:Threshold=70

More advanced usage on whether you want have a threshold line/branch/method in the docs.

[terryaney]

If you only cared about the results posting to Azure DevOps and not the --logger trx, I'd presume you could drop that part of your command perhaps.

Well, I guess I should re-read your post again for the 20th time, lol. I did the --logger trx so that I could post to TFS (I'm not on ADO yet). But I couldn't figure out how to post a 'xUnit' format. I tried just sending the cobertura.xml file in the 'Test Results File' parameter of the Publish Test Results v1 task, but it said no valid file found.

Admittedly, I'm just starting to learn about CICD in TFS, but as you can see below, I have the 'Tests' tab but I don't get a 'Coverage' tab (although I can download the zip file). I'm not sure if that is because of something I'm doing wrong or that TFS version we are on does not support it. I was going to look into it more, but this comment seems to say that if you try publishing *.trx AND a cobertura xml file it can cause issues

[joshjohanning]

Haha! @terryaney, even I had to re-read what I wrote a few times 😁

But that makes sense. I don't think the version of TFS should matter (but it's possible it does).

Ahh, it seems I'm mistaken - when focusing on the Code Coverage report I forgot about the unit test results themselves! The .trx is the test results file. So yes, you want to keep that :). Checking the box in the dotnet task adds this parameter automatically to generate and then it automatically uploads.

You can upload manually, though, with the Publish Test Results task (it sounds like this is what you are maybe doing).

I think instead of using the "Publish Test Results" task to upload the code coverage/cobertura.xml file, you should use the Publish code coverage results task.

      # Publish the combined code coverage to the pipeline
      - task: PublishCodeCoverageResults@1
        displayName: 'Publish code coverage report'
        inputs:
          codeCoverageTool: 'Cobertura'
          summaryFileLocation: '$(Build.SourcesDirectory)/CoverageResults/Cobertura.xml'
          reportDirectory: '$(Build.SourcesDirectory)/CoverageResults'

If you upload the cobertura format, you will get something a little more in-depth, like this that breaks down the coverage percentage for each file (my example isn't the greatest example in the world since I only had 1 file):

I'm not exactly how this will show in your older version of TFS, though, since there is no separate tab for code coverage based on your screenshot 😄. Try it out and let us know!

---

And finally, I don't think I included it in the post, but here's the YAML pipeline this post is using for reference.

I played around with updating the PublishCodeCoverageResults task to the latest version (screenshot), but I don't know, I think I like the code coverage summary generated by ReportGenerator 4.6.1.0 (screenshot) the best since it's the cleanest 🤔 . I'd be tempted to install this version of the CLI and publish that version of the HTML report instead.

Here's an example where I did just that. You have to use PublishCodeCoverageResults@1 and the reportDirectory input, and additionally create a disable.coverage.autogenerate variable and set it to true.

[terryaney]

I'll try posting here my build pipeline (note, I don't know how to get YAML from TFS version) but additionally I run some custom command line stuff I'll mention, but see if you can spot the difference because as is, I don't get code coverage, but I think I'm doing same thing you've mentioned.

Here is summary of my pipeline:

1. Process Project References - Command Line task. Due to the project I'm working on, this task will not doing anything.
2. Restore and Build - Just .NET Core task
   1. Command: build
   2. Project(s): **/*.csproj
   3. Arguments: --configuration Release.
3. Check for Test Projects - PowerShell script that Write-Output "##vso[task.setvariable variable=TestsExists]True" if there are any 'test projects' so remaining 'test' tasks can use the and(succeeded(), eq(variables['TestsExists'], True)) condition determining whether it runs or not (to avoid warnings/errors from tasks b/c of missing expected test output).
4. Test - Just a .NET Core task
   1. Command: test
   2. Project(s): **/tests/**/*.csproj
   3. Arguments: --configuration Release --no-build --logger trx;LogFileName=TestResults.trx /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:ExcludeByFile=**/LoggerMessage.g.cs /p:CoverletOutput=TestResults/coverage.cobertura.xml
5. Generate Test Coverage Reports - Command Line task to run reportgenerator.exe
   1. Working Directory: BUILD_SOURCESDIRECTORY
   2. Arguments: -reports:{buildSourceDirectory}/**/coverage.cobertura.xml -targetdir:{agentTempDirectory}/CodeCoverage -reporttypes:HtmlInline_AzurePipelines;Cobertura
6. Publish Test Results - Standard Publish Test Results v1 task
   1. Test Result Format: VSTest
   2. Test Result Files: **/TestResults.trx
   3. Merge Test Results: true/checked
   4. Test Run Title: Test Results
7. Publish Code Coverage - Standard Publish Code Coverage v1 task
   1. Code Coverage Tool: Cobertura
   2. Summary File: $(Agent.TempDirectory)/CodeCoverage/Cobertura.xml
   3. Report Directory: $(Agent.TempDirectory)/CodeCoverage

So that is my setup. I'm not sure why I don't get a coverage file, but as you saw in the screen shot, I get the Tests tab and it is functional, but I do not get Coverage tab, but below is screen shot of artifacts, which I think looks right as well. If you are able to spot anything explaining why Coverage doesn't work, I'm all ears.

Note: I can download this zip and open the index.html, but I just can't get anything inside of TFS browsable automatically.

[joshjohanning]

Thanks for sharing your pipeline @terryaney, did you ever figure it out?

Is it something like a pathing error? Or that the Cmd line task isn't running correctly since that is a writing to the $(Agent.TempDirectory) that the publish code coverage step is looking for.

If you only had a single test project, you could probably scrap the Generate Test Coverage Reports command line task and instead try to publish the $(Build.SourcesDirectory)/CodeCoverage/Cobertura.xml file directly with the publish code coverage step.

If you have multiple test projects that you do need to combine, you could try one of these debugging steps:

1. add a step to dir (list files) in the $(Agent.TempDirectory)` to see if your Cmd line task is actually creating the report or not
2. In the cmd line task, perhaps save the output into a known spot like $(Build.SourcesDirectory)/reportgenerator so it's published with the build and you can easily see the folder structure
3. I'm not sure if the variables are right in the cmd line task, double check them! Instead of {buildSourceDirectory} you should just be able to use $(Build.SourcesDirectory) and it should translate that for you when you run the step.

[sontambharat]

How can we exclude certain projects from code coverage tool? Currently I am using .runsettings as below but it is not working as expected.
and the test command goes like this

arguments: '--configuration ${{ parameters.buildConfiguration }} --framework ${{ parameters.framework }} --no-restore --collect "XPlat Code Coverage" --settings CodeCoverage.runsettings'

<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage">
        <Configuration>
          <CodeCoverage>
            <ModulePaths>
              <!-- Include assemblies to be analyzed for code coverage -->
              <Include>
                <ModulePath>.*\.dll$</ModulePath>
              </Include>
              <!-- Exclude assemblies from code coverage -->
              <Exclude>
                <ModulePath>Solution.Project.**</ModulePath>
                <!-- Add more <ModulePath> elements to exclude additional assemblies -->
              </Exclude>
            </ModulePaths>
            <!-- Other code coverage settings -->
          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

[joshjohanning]

How can we exclude certain projects from code coverage tool?

Hmm @sontambharat - I might be tempted to use ReportGenerator in this case (example). You could have the code coverage reports created, and then simply delete the project(s) you want to exclude before running the ReportGenerator tool that should then combine them all (minus the excluded ones that were deleted).

[terryaney]

Thanks for sharing your pipeline @terryaney, did you ever figure it out?

Unfortunately, no. I've simplified my build pipeline, but no change.

During the processing in my command line tool for the first step, it simply runs this command:

var testCommand = Cli.Wrap( "dotnet.exe" )
	.WithWorkingDirectory( csProjFile.DirectoryName! )
	.WithArguments( new string[] {
		"test", csProjFile.Name, 
		"--no-build",
		"--configuration", "Release",
		"--logger", "trx;LogFileName=TestResults.trx",
		"/p:CollectCoverage=true",
		"/p:CoverletOutputFormat=cobertura",
		$"/p:Include=[KAT.{string.Join(".", csProjFile.Name.Split('.').TakeUntil( p => p == "Tests", includeCurrent: false ))}*]*",
		"/p:ExcludeByFile=**/LoggerMessage.g.cs",
		"/p:CoverletOutput=TestResults/coverage.cobertura.xml"
	} )

Then I generate coverage reports via:

var reportCommand = Cli.Wrap( variables.ReportGeneratorPath )
	.WithWorkingDirectory( variables.BuildSourceDirectory )
	.WithArguments( new string[] { 
		$"\"-reports:{variables.BuildSourceDirectory}/**/coverage.cobertura.xml\"", 
		$"\"-targetdir:{variables.AgentTempDirectory}/CodeCoverage\"", 
		"-reporttypes:HtmlInline_AzurePipelines;Cobertura" 
	} )!;

But I still only get the following view: