Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add JUnit report logger #458

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add JUnit report logger #458

wants to merge 1 commit into from

Conversation

HypeMC
Copy link
Contributor

@HypeMC HypeMC commented Jan 6, 2024

This pull request adds the ability to generate a JUnit log file for slow tests. Currently, when reviewing a PR, one must go to the CI output to see if there are any slow tests. Now, a JUnit log file can be generated instead and used with GitHub actions:

- name: Publish Slow Test Report
  uses: mikepenz/action-junit-report@v3
  if: always()
  with:
    check_name: PHPUnit Slow Tests Report
    annotate_only: 'true'
    report_paths: ./slow-tests.xml

This makes it easier and more clear to spot slow tests:

image

To trigger the file generation, all that's required is to add the name of the file to phpunit.xml:

<bootstrap class="Ergebnis\PHPUnit\SlowTestDetector\Extension">
  <parameter name="junit-file" value="slow-tests.xml"/>
</bootstrap>

If PHPUnit is run with the --log-junit= option and the junit-file parameter is present, the JUnit log for slow tests is generated.

Later, if needed, other formats such as testdox could be added.

@codecov Codecov
Copy link

codecov bot commented Jan 6, 2024 •
edited
Loading

Codecov Report

Attention: 73 lines in your changes are missing coverage. Please review.

Comparison is base (e55d27d) 69.86% compared to head (6e09889) 67.20%.
Report is 35 commits behind head on main.

Files Patch % Lines
src/Extension.php 0.00% 36 Missing ⚠️
src/Logger/LoggerFactory.php 0.00% 28 Missing ⚠️
src/Reporter/NullReporter.php 0.00% 4 Missing ⚠️
src/Logger/NullLogger.php 0.00% 3 Missing ⚠️
src/FileWriter/DefaultFileWriter.php 88.88% 2 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##               main     #458      +/-   ##
============================================
- Coverage     69.86%   67.20%   -2.66%     
- Complexity      110      155      +45     
============================================
  Files            26       38      +12     
  Lines           521      677     +156     
============================================
+ Hits            364      455      +91     
- Misses          157      222      +65

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

HypeMC
HypeMC commented Jan 6, 2024
View reviewed changes
test/EndToEnd/Version07/Logger/WithConfigurationAndOption/test.phpt
Comment on lines +29 to +40
...... 6 / 6 (100%)<?xml version="1.0" encoding="UTF-8"?>
<testsuites><testsuite name="Slow Tests" tests="6" failures="6" errors="0"><testcase name="Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Version07\Logger\WithConfigurationAndOption\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDuration" file="%stest/EndToEnd/Version07/Logger/WithConfigurationAndOption/SleeperTest.php" line="24"><failure type="slow_test"><![CDATA[The actual duration of 0.5%s exceeds the maximum allowed duration of 0.500.]]></failure></testcase><testcase name="Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Version07\Logger\WithConfigurationAndOption\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider with data set #0 (600)" file="%stest/EndToEnd/Version07/Logger/WithConfigurationAndOption/SleeperTest.php" line="38"><failure type="slow_test"><![CDATA[The actual duration of 0.6%s exceeds the maximum allowed duration of 0.500.]]></failure></testcase><testcase name="Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Version07\Logger\WithConfigurationAndOption\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider with data set #1 (700)" file="%stest/EndToEnd/Version07/Logger/WithConfigurationAndOption/SleeperTest.php" line="38"><failure type="slow_test"><![CDATA[The actual duration of 0.7%s exceeds the maximum allowed duration of 0.500.]]></failure></testcase><testcase name="Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Version07\Logger\WithConfigurationAndOption\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider with data set #2 (800)" file="%stest/EndToEnd/Version07/Logger/WithConfigurationAndOption/SleeperTest.php" line="38"><failure type="slow_test"><![CDATA[The actual duration of 0.8%s exceeds the maximum allowed duration of 0.500.]]></failure></testcase><testcase name="Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Version07\Logger\WithConfigurationAndOption\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider with data set #3 (900)" file="%stest/EndToEnd/Version07/Logger/WithConfigurationAndOption/SleeperTest.php" line="38"><failure type="slow_test"><![CDATA[The actual duration of 0.9%s exceeds the maximum allowed duration of 0.500.]]></failure></testcase><testcase name="Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Version07\Logger\WithConfigurationAndOption\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider with data set #4 (1000)" file="%stest/EndToEnd/Version07/Logger/WithConfigurationAndOption/SleeperTest.php" line="38"><failure type="slow_test"><![CDATA[The actual duration of 1.0%s exceeds the maximum allowed duration of 0.500.]]></failure></testcase></testsuite></testsuites>


Detected 6 tests where the duration exceeded the maximum duration.

1. 1.0%s (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Version07\Logger\WithConfigurationAndOption\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider with data set #4 (1000)
2. 0.9%s (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Version07\Logger\WithConfigurationAndOption\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider with data set #3 (900)
3. 0.8%s (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Version07\Logger\WithConfigurationAndOption\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider with data set #2 (800)
4. 0.7%s (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Version07\Logger\WithConfigurationAndOption\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider with data set #1 (700)
5. 0.6%s (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Version07\Logger\WithConfigurationAndOption\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider with data set #0 (600)
6. 0.5%s (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Version07\Logger\WithConfigurationAndOption\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDuration
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks a lot better on version 10, unfortunately, lower versions don't have a --no-output option.

test/EndToEnd/Version07/Logger/WithNoConfigurationAndOption/test.phpt
Comment on lines +31 to +40
Detected 6 tests where the duration exceeded the maximum duration.

1. 1.0%s (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Version07\Logger\WithNoConfigurationAndOption\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider with data set #4 (1000)
2. 0.9%s (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Version07\Logger\WithNoConfigurationAndOption\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider with data set #3 (900)
3. 0.8%s (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Version07\Logger\WithNoConfigurationAndOption\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider with data set #2 (800)
4. 0.7%s (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Version07\Logger\WithNoConfigurationAndOption\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider with data set #1 (700)
5. 0.6%s (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Version07\Logger\WithNoConfigurationAndOption\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider with data set #0 (600)
6. 0.5%s (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Version07\Logger\WithNoConfigurationAndOption\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDuration

Time: %s, Memory: %s
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lower versions also don't allow the EXPECTF section to be empty.

@localheinz localheinz self-assigned this Jan 15, 2024
@localheinz
Copy link
Member

Apologies for the delay, @HypeMC, I'm taking a look at this now!

@HypeMC
Copy link
Contributor Author

HypeMC commented Apr 8, 2024

@localheinz No problem, I can rebase if needed since there are conflicts, but I don't wanna do it while you're reviewing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ergebnis-bot ergebnis-bot Awaiting requested review from ergebnis-bot ergebnis-bot is a code owner

@localheinz localheinz Awaiting requested review from localheinz localheinz is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@localheinz localheinz

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@HypeMC @localheinz