[RFR] Print test failures in GH Actions #765
Conversation
|
Is this PR (or the other one with a similar name) something to look at already, or is it still work in progress? |
|
Not ready for prime time yet; I'm facing a stumbling block that I figured we could discuss in more detail tomorrow. Specifically, the
The GH Actions logs have nothing to say on the subject, so I'm at a loss. Closed the other PR related to this as I think the approach here is more likely to be useful. |
|
I think GHA doesn't run step n+1 if step n has failed, so you might want to either make the "Run tests" step exit with a zero error code, merge the "Run tests" and "Print test failures" into a single step, or (perhaps better approach) add some conditionals. Also, running the full test suite takes a whole lot of time so you might consider temporarily removing a bunch of tests from the codebase just to make the GHA run finish faster. Later you can just |
|
Finally got this working as intended—see the "List errors and test failures" step in this build |
jtotoole
changed the title
|
ready for review! |
|
Merged in |
|
So printing failures as GitHub Actions annotations didn't work? |
|
I initially abandoned that approach because the GitHub Checks API will only publish annotations on test files that have been changed in the PR. There's a marketplace action that will annotate everywhere if we first create a test summary file in JUnit XML format, but that seemed like rather high-hanging fruit compared to this approach. That said, I'm happy to give the JUnit thing a shot if you think it'd be better and not wildly complicated. |
jtotoole
changed the title
| @@ -189,3 +189,7 @@ jobs: | |||
| MC_TWITTER_ACCESS_TOKEN_SECRET: ${{ secrets.MC_TWITTER_ACCESS_TOKEN_SECRET }} | |||
| MC_TWITTER_CONSUMER_KEY: ${{ secrets.MC_TWITTER_CONSUMER_KEY }} | |||
| MC_TWITTER_CONSUMER_SECRET: ${{ secrets.MC_TWITTER_CONSUMER_SECRET }} | |||
|
|
|||
| - name: List test failures and errors | |||
| run: echo -e 'Test failures:\n' && awk '/FAILURES/,/short/' /home/runner/runners/*/_diag/pages/*.log && echo -e '\n\nErrors:\n' && awk '/non-zero/,/status 1/' /home/runner/runners/*/_diag/pages/*.log | |||
There was a problem hiding this comment.
What's in /home/runner/runners/*/_diag/pages/*.log, i.e. how are you finding those errors? Do you perhaps have a sample log file from that directory that we could look at (without various private credentials that might be in this log)?
Grepping (awking?) out failures from a log file is a bit wonky because any test might print a string FAILURES at any point for a legitimate reason and still succeed; plus, tests don't have to fail with status 1 specifically as they can have any non-zero exit status.
No description provided.