How does SFDX-hardis handle code coverage reports in deploy and test commands?

[mcarvin8]

Hi. I have a quick question regarding how sfdx-hardis handles code coverage reports generated by the sf project deploy and sf apex test commands from the CLI.

I know sfdx-hardis uses both CLI commands to execute deployments/tests in the org. So I'm guessing hardis is limited to what code coverage reports are currently supported by the Salesforce CLI, am I correct?

Something I've seen in various CLI issues/discussions

1. The 2 CLI commands, test and deploy, produce different coverage formats. For example, even if you use the "json" format for both commands, you'll see slightly different formatting.
2. The CLI deploy command produces inaccurate "covered" lines or lines in general. i.e. line 120 in a 100-line apex class is returned as "covered" or 120 lines in total are reported back for a 100-line class. This bug has been active for years and only addressable by the server team. Apparently, they only work with support case tickets if a lot of users create cases.
3. Apex class paths are hard-coded with the no-map prefix in the CLI code-base
4. Different coverage formats between the deploy and test commands. for example, you can make cobertura reports with deploy, but not with test command.

I wrote a CLI plugin that can handle the different coverage formats from the CLI deploy and test command (just using the standard "json" format provided by both), "fix" the deploy coverage inaccuracies, and add correct file-paths per your Salesforce DX project (using the sfdx-project.json file for package directories).

https://github.com/mcarvin8/apex-code-coverage-transformer

Not sure if you already have code in hardis that can handle some of these code coverage cases, but my plugin can also handle creating 4 possible formats at the moment: sonarqube, cobertura, clover, and lcovonly.info

If you want to discuss incorporating my plugin into hardis for advanced code coverage reporting so users can use sfdx-hardis to create code coverage reports for external tools like SonarQube, GitLab (Cobertura), etc., let me know.

That way, if users use the hardis deploy or test commands, they can get accurate code coverage reports they can use elsewhere for code quality.

[nvuillam]

@mcarvin8 thanks for your interest in sfdx-hardis :)

About coverage reports, to be honest for the moment we only handle the fact that it is > to a percentage, and it's currently not very documented (but thanks to your message I will enhance that ^^ )

The coverage check is configurable these ways:

• apexTestsMinCoverageOrgWide in .sfdx-hardis.yml config file
• APEX_TESTS_MIN_COVERAGE_ORG_WIDE environment variable

If not on these values is set, the default is 75.0% (it is recommended to update it when doing sfdx-hardis CI/CD setup)

This configuration is used in the following commands:

• https://sfdx-hardis.cloudity.com/hardis/org/test/apex/

  • makes the comand fail is coverage < N%

• https://sfdx-hardis.cloudity.com/hardis/project/deploy/quick/ (wrapper around the official SF Cli command)

• https://sfdx-hardis.cloudity.com/hardis/project/deploy/start/ (wrapper around the official SF Cli command)

• https://sfdx-hardis.cloudity.com/hardis/project/deploy/validate/ (wrapper around the official SF Cli command)

  • makes the comand fail is coverage < N%
  • post the failing percentage in the Pull Request deployment status comment

• https://sfdx-hardis.cloudity.com/hardis/project/deploy/smart/ (smart deployment command, the base of sfdx-hardis CI/CD deployment checks and execution)

apex test run command uses arguments --code-coverage --result-format human
Deployment commands require --coverage-formatters json-summary

Does your plugin work with those formats (maybe just the --coverage-formatters json-summary) ?

Do you mean that with your plugin we could generate reports that would show covered/uncovered lines within native GitHub, Gitlab, Azure, Bitbucket ?

If so, it could indeed be interesting, and if you use the generated output (or even a fixed file name that hardis could generate and always put in the same place), we could add an option in pipelines to call your plugin to postprocess those files then send the results to Git services :)

[mcarvin8]

@nvuillam - thanks for getting back.

My plugin currently just reads the json coverage report format from both the CLI test and deploy command since that's one of the shared formats. I prefer to stick with that format since I have type guards for both to automate parsing and it's nice to use just 1 format from both commands to transform.

You can add multiple coverage formats to the Salesforce CLI commands, so if Hardis is using human/json-summary, it can be update to also add json to the command like --coverage-format json json-summary. Perhaps if Hardis has a flag to opt into using my plugin, that flag just appends the json format to the CLI command.

Yea, the Cobertura format my plugin creates allows you to visualize code coverage directly in GitLab, the SonarQube format in SonarQube. I think Cobertura gets availability in most platforms like GitHub, etc. but I have the 3 other formats supported on my end. The goal being transforming what json coverage report Salesforce CLI creates into something another platform can use to visualize coverage in a Salesforce DX project.

My plugin does use a config file to allow a CLI hook, or the commands can be ran manually with the CLI.

[nvuillam]

What if I manage to use --coverage-format json under the hood in both commands, then store the CLI output in a fixed file name, for example hardis-report/coverage-results.json , so you can catch the file and make your magic with it in the next command of the workflow ? :)

[mcarvin8]

That would work perfectly!

[nvuillam]

@mcarvin8 Handled in #985 , and released in 5.14.1 :)

Now it's your turn ;)

[mcarvin8]

#986

[mcarvin8]

Gonna close this as resolved based on the hardis updates being done.

I need to work on updating my existing postrun.ts hook to work with sfdx-hardis without duplicating itself, i.e. running twice since this is looking for the standard sf commands.

But the work on this side is complete.

[mcarvin8]

Local test
Ensure sf cli plugins apex-code-coverage-transformer and sfdx-hardis are installed

Run sf hardis project deploy start command to see if the acc-transformer hook is still picked up in the current state when the .apexcodeoveragetransformer.config.json is found and hardis runs the sf project deploy command.

Confirmed that my sf project deploy start hook still runs when invoked from hardis, which means I need to figure out how to update my hook when invoked from hardis to only run when the hardis command is done.

[mcarvin8]

See mcarvin8/apex-code-coverage-transformer#84. This is probably the path of least resistance. I want to keep the hook as an optional thing for users, so they have to create the config JSON and then set the paths to the pre-defined Hardis path. Instructions posted in my README for those using both plugins.

[mcarvin8]

2.6.0

[nvuillam]

Looks good to me :)
Would be even better with a full example of files when using both plugins together:

• workflow sample (ex: github action)
• .apexcodecovtransformer.config.json