Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use methods.setup_process_afterscript() for process logs #76

Merged
merged 3 commits into from
Jun 3, 2024

Conversation

nwiltsie
Copy link
Member

@nwiltsie nwiltsie commented Jun 3, 2024

Description

This PR updates the pipeline to use the new setup_process_afterscript() method from uclahs-cds/pipeline-Nextflow-config#64.

Doing so allows us to get rid of a ton of boilerplate - specifically, the output: path(".command.*") directive and the corresponding publishDir directive on every process. Now the only required configuration is adding (e.g.) ext log_dir_suffix: { "-${sample_id}" } to every process that wants to customize the output log folder.

Testing Results

  • NFTest
    • log: /hot/software/pipeline/pipeline-recalibrate-BAM/Nextflow/development/unreleased/nwiltsie-streamlined-logging/log-nftest-20240603T210115Z.log
    • cases: default set

Checklist

  • I have read the code review guidelines and the code review best practice on GitHub check-list.

  • I have reviewed the Nextflow pipeline standards.

  • The name of the branch is meaningful and well formatted following the standards, using [AD_username (or 5 letters of AD if AD is too long)]-[brief_description_of_branch].

  • I have set up or verified the branch protection rule following the github standards before opening this pull request.

  • I have added my name to the contributors listings in the manifest block in the nextflow.config as part of this pull request, am listed
    already, or do not wish to be listed. (This acknowledgement is optional.)

  • I have added the changes included in this pull request to the CHANGELOG.md under the next release version or unreleased, and updated the date.

  • I have updated the version number in the metadata.yaml and manifest block of the nextflow.config file following semver, or the version number has already been updated. (Leave it unchecked if you are unsure about new version number and discuss it with the infrastructure team in this PR.)

  • I have tested the pipeline using NFTest, or I have justified why I did not need to run NFTest above.

Copy link

github-actions bot commented Jun 3, 2024

Bleep bloop, I am a robot.

Alas, some of the Nextflow configuration tests failed!

test/configtest-F16.json

@ ["process","afterScript"]
+ {"1":"","2":"","3":"","closure":""}
@ ["process","ext"]
+ {"capture_logs":true,"commonAfterScript":{"1":"","2":"","3":"","closure":""},"log_dir":{"1":"ext","2":"ext","3":"ext","closure":"ext"}}

test/configtest-F32.json

@ ["process","afterScript"]
+ {"1":"","2":"","3":"","closure":""}
@ ["process","ext"]
+ {"capture_logs":true,"commonAfterScript":{"1":"","2":"","3":"","closure":""},"log_dir":{"1":"ext","2":"ext","3":"ext","closure":"ext"}}

If the above changes are surprising, stop and determine what happened.

If the above changes are expected, there are two ways to fix this:

  1. Automatically: Post a comment starting with "/fix-tests" (without the quotes) and I will update the tests for you (you must review my work afterwards).
  2. Manually: Follow these steps on Confluence.

@nwiltsie
Copy link
Member Author

nwiltsie commented Jun 3, 2024

/fix-tests

Copy link

github-actions bot commented Jun 3, 2024

Bleep bloop, I am a robot.

I have updated all of the failing tests for you with 328f39d. You must review my work before merging this pull request!

Copy link
Collaborator

@yashpatel6 yashpatel6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@nwiltsie nwiltsie merged commit 1e23e07 into main Jun 3, 2024
7 checks passed
@nwiltsie nwiltsie deleted the nwiltsie-streamlined-logging branch June 3, 2024 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants