Secrets not working with bash .command.run

[nick-youngblut]

Bug report

Expected behavior and actual behavior

AFAIK, to debug a process, one normally moves to the process working directory and runs bash .command.run.

At least with Nextflow 24.10.2, this results in grep: /home/nickyoungblut/.nextflow/secrets/.nf-0f268357-4937-450a-a7ec-9f343ae363e9.secrets: No such file or directory, and the secret is not provided for the process.

I believe that for previous versions of Nextflow, the secret was provided to the process script when directly calling bash .command.run.

Also, in regards to the docs:

This feature is only available when using the local or grid executors (Slurm, Grid Engine, etc). The AWS Batch executor allows the use of secrets when deploying the pipeline execution via Seqera Platform.

There is no mention of GCP. Must one also use the Seqera Platform to utilize secrets if using GCP (instead of AWS)? a

Steps to reproduce the problem

Set a secret via nextlfow secret set my_secret my_secret_value and use it in a process via secret "my_secret". When the process is executed via nextflow run, the secret is provided. However, the secret is not provided if running bash .command.run.

Program output

Environment

• Nextflow version: 24.10.2
• Java version: 11.0.1
• Operating system: 22.04.4
• Bash version: 5.1.16

[nick-youngblut]

According to the .command.run file:

source /dev/stdin <<<"$(cat <(grep -w -e 'my_secret=.*' /home/nickyoungblut/.nextflow/secrets/.nf-4aa05d79-d6d0-464b-a166-b01307897573.secrets))"

/home/nickyoungblut/.nextflow/secrets/ only contains store.json.

The .command.run file is incorrectly assuming that *.secrets exists.

[nick-youngblut]

Is .nextflow/secrets/.nf-*.secrets deleted right after the job run finishes? If so, how could one actually debug any job that includes a secret, as noted in the standard Nextflow job failure message:

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh

The .command.sh script does not create the .nextflow/secrets/.nf-*.secrets. It just looks for the existing file via:

source /dev/stdin <<<"$(cat <(grep -w -e 'my_secret=.*' /home/nickyoungblut/.nextflow/secrets/.nf-4aa05d79-d6d0-464b-a166-b01307897573.secrets))"

[pditommaso]

You are right, this could be a drawback of this change e28537ca. We need to revert it