Singularity image is not working in nextflow

Question

[optimistsso4co3]

Hello everyone,

I'm having trouble getting Singularity to work with my custom bcftools pipeline in Nextflow.

I've tested my pipeline with Conda and it works great. However, when I install software locally (instead of using Conda), it runs approximately 10 times faster due to the large number of times the Conda environment needs to be invoked.

To test Singularity, I followed these steps:

1. downloaded bcftools singularity image from: https://cloud.sylabs.io/library/weizhu365/mocca-sv/bcftools_1-9
2. tested it with singularity run bcftools_1-9_1.0.0.sif
3. added necessary lines to nextflow.config:

   process.container = '/path/to/bcftools_1-9_1.0.0.sif'
   singularity.enabled = true
   

However, it doesn't seem to recognize the image. I've successfully run other Nextflow pipelines with Singularity, such as nf-core's nextflow run nf-core/sarek -profile test,singularity -r 2.7.1.

Do I need to take any additional steps? Any help would be greatly appreciated.

[ATpoint]

It should work. At least this works:

$ cat main.nf 
nextflow.enable.dsl = 2

process Test {

  output:
  stdout

  script:
  """
  bcftools --version
  """

}

workflow { Test() }  

$ cat nextflow.config 
process.container = "/home/a/a_toen03/bcftools_1.17--haef29d1_0.sif"
singularity.enabled = true

...correctly running the bcftools command. Please elaborate on what exactly goes wrong. What is the error message (if any)? Can you try with the container I linked? Get via singularity pull docker://quay.io/biocontainers/bcftools:1.17--haef29d1_0 from quay.

[optimistsso4co3]

The command bcftools --version is functioning properly. However, when attempting to use the command bcftools view -h file.vcf.gz to view a specific file in the current directory, an error message is generated indicating that the file does not exist, even though it's in the working directory. In fact, no file can be detected when singularity image is specified.

Can you please test if this issue is present in your system?

My singularity version: 3.4.1-4.2.ohpc.1.3.9

[Darked89]

So the issue is that the container somehow does not have an access to file.vcf.gz? If you specified file.vcf.gz as nextflow's input you should have a softlink to that file in the nextflow work subdirectory. You may just get into the container singularity shell your_container.sif and check what is visible from there (ls, df)

[optimistsso4co3]

I tested singularity shell your_container.sif inside the Work subdirectory and, even though file.vcf.gz is visible (ls -l), it can not be red when your_container.sif is activated

less: can't open, no such file or dir

On the other hand, process files (e.g. .command.sh) can be red

[ATpoint]

That's not a helpful diagnostic as you need to mount folders if running singularity manually -- something that nextflow does for you. Again, please provide a reproducible example that includes some minimal testing data and the Nextflow code. Singularity is used with Nextflow by thousands of users, so it must be something with how you read/stage/provide the input data.

[ATpoint]

Please provide a reproducible example for debugging.