Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove local set_env and check_limits methods #46

Merged
merged 2 commits into from
Dec 21, 2023
Merged

Conversation

nwiltsie
Copy link
Member

@nwiltsie nwiltsie commented Dec 20, 2023

Description

This just removes the local copies of methods.set_env and methods.check_limits in favor of the updated versions from the submodule.

Unfortunately there's not a great way to show that the code blocks are equivalent other than by pasting diffs in here (I stripped a few comments):

--- local_check.config	2023-12-20 15:50:02.115400000 -0800
+++ submodule_check.config	2023-12-20 15:50:01.447858000 -0800
@@ -1,4 +1,4 @@
-    check_limits = { obj, type ->
+    check_limits = { Object obj, String type ->
         if (type == 'memory') {
             try {
                 if (obj.compareTo(params.max_memory as nextflow.util.MemoryUnit) == 1)
@@ -8,7 +8,7 @@
                 else
                     return obj
             } catch (all) {
-                println "   ### WARNING ###   Max memory '${params.max_memory}' or min memory '${params.min_memory}' is not valid! Using default value: $obj"
+                System.out.println("   ### WARNING ###   Max memory '${params.max_memory}' or min memory '${params.min_memory}' is not valid! Using default value: $obj")
                 return obj
             }
         } else if (type == 'time') {
@@ -20,14 +20,14 @@
                 else
                     return obj
             } catch (all) {
-                println "   ### WARNING ###   Max time '${params.max_time}' or min time '${params.min_time}' is not valid! Using default value: $obj"
+                System.out.println("   ### WARNING ###   Max time '${params.max_time}' or min time '${params.min_time}' is not valid! Using default value: $obj")
                 return obj
             }
         } else if (type == 'cpus') {
             try {
-                return Math.max( Math.min( obj, params.max_cpus as int ), params.min_cpus as int )
+                return Math.max( Math.min( obj as int, params.max_cpus as int ), params.min_cpus as int )
             } catch (all) {
-                println "   ### WARNING ###   Max cpus '${params.max_cpus}' or min cpus '${params.min_cpus}' is not valid! Using default value: $obj"
+                System.out.println("   ### WARNING ###   Max cpus '${params.max_cpus}' or min cpus '${params.min_cpus}' is not valid! Using default value: $obj")
                 return obj
             }
         }
--- local_set_env.config	2023-12-20 15:52:05.911211000 -0800
+++ submodule_set_env.config	2023-12-20 15:52:06.585748000 -0800
@@ -1,13 +1,17 @@
     set_env = {
         if (params.ucla_cds) {
-            params.work_dir = (params.containsKey('work_dir') && params.work_dir) ? params.work_dir : '/scratch'
-            if (methods.check_workdir_permissions(params.work_dir)) {
-                workDir = params.work_dir
-            }
+            params.work_dir = (params.containsKey('work_dir') && params.work_dir) ? params.work_dir : methods.get_default_workdir()
+            schema.check_path(params.work_dir, 'w')
+            workDir = params.work_dir
         } else {
             // If work_dir was specified as a param and exists or can be created, set workDir. Otherwise, let Nextflow's default behavior dictate workDir
-            if (params.containsKey('work_dir') && params.work_dir && methods.check_workdir_permissions(params.work_dir)) {
+            // Default Nextflow behavior: NXF_WORK environment variable if set, otherwise "${launchDir}/work"
+            if (params.containsKey("work_dir") && params.work_dir) {
+                schema.check_path(params.work_dir, 'w')
                 workDir = params.work_dir
+            } else {
+                // Set the work_dir to match the default Nextflow behavior for parity
+                params.work_dir = System.getenv("NXF_WORK") ?: "${launchDir}/work"
             }
         }
     }

Testing Results

  • NFTest
    • log: /hot/software/pipeline/pipeline-recalibrate-BAM/Nextflow/development/unreleased/nwiltsie_fix_scratch/log-nftest-20231220T223613Z.log
    • cases: default set

Checklist

  • I have read the code review guidelines and the code review best practice on GitHub check-list.

  • I have reviewed the Nextflow pipeline standards.

  • The name of the branch is meaningful and well formatted following the standards, using [AD_username (or 5 letters of AD if AD is too long)]-[brief_description_of_branch].

  • I have set up or verified the branch protection rule following the github standards before opening this pull request.

  • I have added my name to the contributors listings in the manifest block in the nextflow.config as part of this pull request, am listed
    already, or do not wish to be listed. (This acknowledgement is optional.)

  • I have added the changes included in this pull request to the CHANGELOG.md under the next release version or unreleased, and updated the date.

  • I have updated the version number in the metadata.yaml and manifest block of the nextflow.config file following semver, or the version number has already been updated. (Leave it unchecked if you are unsure about new version number and discuss it with the infrastructure team in this PR.)

  • I have tested the pipeline using NFTest, or I have justified why I did not need to run NFTest above.

@yashpatel6
Copy link
Collaborator

@nwiltsie I ended up removing the local set_env in #45 so the branch here will need to be updated to resolve the conflicts

@nwiltsie
Copy link
Member Author

@yashpatel6 You monster. Changes merged, I'm re-running NFTest for sanity now and I'll report back once that passes.

@nwiltsie
Copy link
Member Author

Confirmed, be57078 (after merging the changes from #45) still passes NFTest: /hot/software/pipeline/pipeline-recalibrate-BAM/Nextflow/development/unreleased/nwiltsie_fix_scratch/log-nftest-20231221T185448Z.log

Copy link
Collaborator

@yashpatel6 yashpatel6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@nwiltsie nwiltsie merged commit 0faae57 into main Dec 21, 2023
1 check passed
@nwiltsie nwiltsie deleted the nwiltsie_fix_scratch branch December 21, 2023 21:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants