Source other conda environments in a nextflow pipeline when nextflow itself is in a conda environment?

Question

[chaco001]

Hello,

I work in an HPC environment which uses slurm.

I am writing nextflow pipelines. Right now, nextflow itself has been installed in a conda environment. However, some of the processes in my nextflow pipeline need to use a different conda environment, because of dependency conflicts.

I have two related questions:

1. How can I have processes use a conda environment that differs from the environment in which nextflow is being run?

2. Have I set this up wrong in the first place and is it better to have nextflow outside of the conda environment? If so, what's the best way to distribute the whole pipeline along with nextflow?

Thank you!

[Arup Ghosh]

Use the standalone version of Nextflow and specify the process-specific conda environments paths.

Answer 1

[colindaven]

I used to use nextflow in a conda env. That requires the env to be active when a pipeline is started. Also I have productive pipelines which require fully isolated nextflow, so that would mean an individual conda env per pipeline. That is slow to create and includes a lot of small files.

That is why I use the nextflow single jar distribution and a globally located java17 on an NFS storage. The jar is very quick to download.

I use singularity containers created from a Dockerfile for the pipeline dependencies, as they tend to be more static and reliable than conda envs. Micromamba can be easily used for conda packages within a Docker/Sing. container.

You can set containers or conda per process in the nextflow main.nf file, or -better - in the nextflow.config file in a process block (along with RAM, cpus and retry). Isolating the hardware config from the code into the config file is the way to go.

Answer 2

[ATpoint]

You can make each process use a dedicated environment, see https://www.nextflow.io/docs/latest/conda.html#use-existing-conda-environments

The nextflow binary via conda is fine. No preference here. Or just install Nextflow as described in the Nextflow docs. It doesn't matter.