FATAL: While making image from oci registry: while building SIF from layers: conveyor failed to get when pulling a Singularity container

[b-lac]

Description of the bug

Hello,

After launching the pipeline, it fails at "STAR_GENOMEGENERATE" because a container fails to get pulled.
I checked that I have enough space on disk for the singularity cache and library.

Command used and terminal output

command is:

nextflow run \
    nf-core/rnaseq \
    -r 3.18.0 \
    -profile singularity \
    -c "nextflow_conf/slurm.config" \
    --input $USER/cohort-rna-samplesheet.4.csv \
    --outdir $RESULTS/results_rna \
    --gtf $HOME/hg38broad/hg38_v0_star_v2_7_10a_modified_v43.annotation.gtf \
    --fasta $HOME/hg38broad/hg38.fasta \
    --igenomes_ignore \
    --genome null

and output is :

Error executing process > 'NFCORE_RNASEQ:PREPARE_GENOME:STAR_GENOMEGENERATE'

Caused by:
  Failed to pull singularity image
    command: singularity pull  --name quay.io-nf-core-htslib_samtools_star_gawk-311d422a50e6d829.img.pulling.1737454143217 docker://quay.io/nf-core/htslib_samtools_star_gawk:311d422a50e6d829 > /dev/null
    status : 255
    hint   : Try and increase singularity.pullTimeout in the config (current is "1h")
    message:
      time="2025-01-21T11:09:03+01:00" level=warning msg="\"./xdg-runtime-dir\" directory set by $XDG_RUNTIME_DIR does not exist. Either create the directory or unset $XDG_RUNTIME_DIR.: stat ./xdg-runtime-dir: no such file or directory: Trying to pull image in the event that it is a public image."
      �[34mINFO:   �[0m Starting build...
      time="2025-01-21T11:09:13+01:00" level=warning msg="\"./xdg-runtime-dir\" directory set by $XDG_RUNTIME_DIR does not exist. Either create the directory or unset $XDG_RUNTIME_DIR.: stat ./xdg-runtime-dir: no such file or directory: Trying to pull image in the event that it is a public image."
      �[31mFATAL:  �[0m While making image from oci registry: while building SIF from layers: conveyor failed to get: Error reading blob sha256:32b824d45c6101d459f5d3c13ab8658b6f79713f3bd64e363d3f6bc98faf5d6d: Get https://cdn03.quay.io/quayio-production-s3/sha256/32/32b824d45c6101d459f5d3c13ab8658b6f79713f3bd64e363d3f6bc98faf5d6d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAI5LUAQGPZRPNKSJA%2F20250121%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20250121T100914Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=a9c7ca16e66acaa2f5ffcfb718eedb6cbdad904b33c80a3eac62ec1d38f736dc&cf_sign=kPDzEda9SnPL%2F8jKkYCseI3wpq6DZcRtdCTaBoojgIbv5GRyLFdAqgmr%2B%2FAhH4KQkpHTIZqbBx6CKVU2C1p6RsYhBy%2Bc%2F%2BNtWsoR46GgZIiZWYXD1ZwqZvr94N24B3m4dVgK5Q%2BJu%2FKGSGikjF%2B9Jhw9Enq6AKjynnr7opc3CQcQQlVBI055L57M7qi3mnS9XZR0HPA1ANIpF5GrpG%2BrjzuTM0uFeCmDU2VATVH3dwCnxndjsXy4iv%2B0ksf55oCeoGiy0jKITaPKuEQOyHZDSbijoCoqBh1blzHm6BtcjhgTBSeEqQvCDPmGIK51bcqe%2Fr6jdlkoCtCSzVxncRwmmA%3D%3D&cf_expiry=1737454754®ion=us-east-1&namespace=nf-core&repo_name=htslib_samtools_star_gawk: read tcp 10.38.45.1:49668->172.64.150.109:443: read: connection reset by peer

Relevant files

.nextflow.log

System information

Nextflow : 24.04.4
Hardware: HPC
Executor: SLURM
Container engine: Singularity
OS: Red Hat
nf-core/rna-seq: 3.18.0

[IrinaVKuznetsova]

I have the same issue

Jan-31 13:24:15.206 [Actor Thread 114] DEBUG nextflow.Session - Session aborted -- Cause: java.lang.IllegalStateException: Failed to pull singulari>
  command: singularity pull  --name nf-core-htslib_samtools_star_gawk-311d422a50e6d829.img.pulling.1738301050236 docker://nf-core/htslib_samtools_s>
  status : 255
  hint   : Try and increase singularity.pullTimeout in the config (current is "1h")
  message:
    WARNING: Couldn't use cached digest for registry: HEAD https://index.docker.io/v2/nf-core/htslib_samtools_star_gawk/manifests/311d422a50e6d829:>
    WARNING: Falling back to direct digest.
    FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://nf-core/htslib_samtools_star_>

I attempted to increase the pullTimeout, but it didn't resolve the issue.

nextflow 24.04.3
nf-core/rnaseq 3.18.0
Linux
-profile singularity

[b-lac]

I have the same issue

Jan-31 13:24:15.206 [Actor Thread 114] DEBUG nextflow.Session - Session aborted -- Cause: java.lang.IllegalStateException: Failed to pull singulari>
  command: singularity pull  --name nf-core-htslib_samtools_star_gawk-311d422a50e6d829.img.pulling.1738301050236 docker://nf-core/htslib_samtools_s>
  status : 255
  hint   : Try and increase singularity.pullTimeout in the config (current is "1h")
  message:
    WARNING: Couldn't use cached digest for registry: HEAD https://index.docker.io/v2/nf-core/htslib_samtools_star_gawk/manifests/311d422a50e6d829:>
    WARNING: Falling back to direct digest.
    FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://nf-core/htslib_samtools_star_>

I attempted to increase the pullTimeout, but it didn't resolve the issue.

nextflow 24.04.3 nf-core/rnaseq 3.18.0 Linux -profile singularity

I think my issue is not related to a bug but to the configuration of my server which might be blocking fetchin containers from quay.io. I have read: connection reset by peer at the end of my error log. It seems you've got a failed to get checksum which might indicate a different reason.

[IrinaVKuznetsova]

Have you managed to resolve your issue where the server isn't configured to fetch an image from quay.io? I think I am facing a similar problem where my server can't fetch Docker images. Waiting on a response from the server admin to fix this or any insights would be appreciated!

[b-lac]

I made a request to unlock the access on my server but I'm still waiting ofr their answer.

Maybe you can try a simple wget or curl request on the URL of your container, and compare results when you are either on your server or on a public network.