Nextflow using existing Conda Environments

Question

[Barry Digby]

Hi,

I am attempting to use an existing conda environment in a nextflow script (I do not want to point to a .yml file and compile environments). I have followed the instructions suggested by nextflow for using an existing environment:

process foo {
conda '/path/to/an/existing/env/directory'

'''
your_command --here
'''
}

Here is my attempt:

#!/usr/bin/env nextflow

process foo {
echo true
conda '/home/barry/anaconda3/envs/circexplorer2'

script:
"""
CIRCexplorer2 --help
"""
}

It gives the folowing error, pointing to the .command.run line 77:

.command.run: line 77: activate: No such file or directory

Looking at the file:

# conda environment
source activate /home/barry/anaconda3/envs/circexplorer2

It seems strange that it is using the depreciated source activate call, but maybe I'm missing a trick. Any thoughts?

p.s I have stripped the path and provided conda 'circexplorer2' to the nextflow script, but it tries to create the environment which is not the behaviour I was hoping for.

Regards,

Barry

[bruce.moran]

Maybe you saw this Github issue?

If I want to use conda as per your example, because the HPC I am on uses Modules, I need to: module load conda, then initialise: conda init bash and then reload the shell: exec bash (this gets a (base) bmoran@cluster:$ prompt). Then I can specify as you want.

I found it easier to create the conda env in a Singularity container from an env.yaml, these build so fast it is rarely an inconvenience, and I think having the container frozen, and potentially available on Singularity Hub/Library for posterity, is useful.

[Barry Digby]

Hi Bruce,

Yep, I derived the local solution (answer below) from a suggestion on the github page linked. Cool, I did not know you could create a container from a .yml file, could you share the singularity call used to build the container? I am keen on making this pipeline reproducible , +1 for posterity.

[bruce.moran]

Based on a Github repo with my_env.yaml therein; environment section is called every time the container executes, so that is activating the env; you can pull the centos7_conda container to see what's in there, just installs miniconda, used to install that in 'project' containers but more efficient to have a 'base' container IMO.

Bootstrap:library
From:bruce.moran/default/bases:centos7_conda

%environment

  source /opt/miniconda/etc/profile.d/conda.sh
  conda activate my_env

%post

  source /opt/miniconda/etc/profile.d/conda.sh
  git clone https://github.com/brucemoran/my_env
  conda env create -n my_env -f ./my_env/my_env.yaml
  conda clean -y -a

[Barry Digby]

Hi Bruce,

I've had a spot of bother making the image. I have replaced the %post with my github repo but singularity build returns ERROR: 'Bootstrap' type not supported: library. I also went to the page where the container is hosted and tried to install via singularity pull, but it returned ERROR: pull is only supported for shub URIs.

[bruce.moran]

Check singularity --version, then update your Singularity version=D

N.B. the code chunk in my example needs to be saved entirely into a recipe.container_name file, then you can use singularity build container_name.sif recipe.container_name to build it and push to https://cloud.sylabs.io/library, or build on the Singularity Hub. During building the env.yaml is cloned from Github to provide the container with the conda env.

Answer 1

[Barry Digby]

Local solution:

sudo ln -s /home/barry/anaconda3/bin/activate /usr/bin/activate

HPC solution :

export PATH="/home/bdigby/Anaconda3/4.4.0/bin/:$PATH"

add to ~/.bashrc