Problem with Mamba

Question

[Begonia_pavonina]

I have an issue with Mamba, it seems to run very well on a single node, but when run in a slurm script, it does not recognize the arguments and gives an error message. Below, an example from a Snakemake run. Did it happen to anyone else? Any suggestions what can be wrong here?

Building DAG of jobs...
Creating conda environment https://github.com/snakemake/snakemake-wrappers/raw/v2.6.0/bio/bwa/index/environment.yaml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /tmp/tmichel_12774347/tmpoorpm5cp.yaml:
Command:
mamba env create --quiet --file "/mnt/shared/scratch/usr/ROH_pipeline/Begonia_full_genome/.snakemake/conda/1e394e56e793f957129208ab3debae09_.yaml" --prefix "/mnt/shared/scratch/usr/ROH_pipeline/Begonia_full_genome/.snakemake/conda/1e394e56e793f957129208ab3debae09_"
Output:
usage: mamba [-h] [--version] [--slow SLOW] [--enable-coverage]
             [--coverage-file COVERAGE_FILE] [--format FORMAT] [--no-color]
             [--tags TAGS]
             [specs ...]
mamba: error: unrecognized arguments: --quiet --file /mnt/shared/scratch/usr/ROH_pipeline/Begonia_full_genome/.snakemake/conda/1e394e56e793f957129208ab3debae09_.yaml --prefix /mnt/shared/scratch/usr/ROH_pipeline/Begonia_full_genome/.snakemake/conda/1e394e56e793f957129208ab3debae09_

[Jeremy Leipzig]

i bet there is some script in your .bashrc (or the one you inherited from root) that is putting you in a hostile conda environment when snakemake dispatches to that worker node.

i would try ssh'ing directly to any node and try to run some commands: mamba env create --quiet --name testenv

[rpolicastro]

I'm adding 'hostile conda environment' to my normal lexicon.

[Ram]

Can you check if the conda/mamba executables you use are the same on both the node you ran things successfully on and on the compute node?

Also, your sysadmin is in a much better position to help you than we are.

Answer 1

[Begonia_pavonina]

I have solved my problem just forcing the use of conda with Snakemake. But out of curiosity I have made some test, and found slightly different version of conda in different nodes.

Test of the version of the software:

Single node:

conda -V
conda 23.3.1
mamba -V
conda 23.3.1

slurm script:

conda -V
conda 23.9.0
mamba -V
conda 23.3.1

Try to make a new environment:

mamba env create --quiet --name testenv
EnvironmentFileNotFound: '/mnt/shared/home/usr/environment.yml' file not found

In the .bashrc file:

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/usr/miniforge3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/home/usr/miniforge3/etc/profile.d/conda.sh" ]; then
        . "/home/usr/miniforge3/etc/profile.d/conda.sh"
    else
        export PATH="/home/usr/miniforge3/bin:$PATH"
    fi
fi
unset __conda_setup

if [ -f "/home/usr/miniforge3/etc/profile.d/mamba.sh" ]; then
    . "/home/usr/miniforge3/etc/profile.d/mamba.sh"
fi
# <<< conda initialize <<<

[gernophil]

There have been some issues in the combination of conda and mamba with conda version 23.5 (and maybe lower). You just might need to update or reinstall a recent version, of possible. I don’t find the GitHub thread anymore, but I will look tomorrow. IIRC it was at the same time as the introduction of the new libmamba-solver for conda.

[Ram]

Yeah, I ran into an issue with a similar background as well. Time to switch to miniforge?

OP seems to be running into a different sort of problem where their login node and compute node are seeing different versions of conda. Probably something to do with OP's profile files.

[gernophil]

I just installed the new miniconda and it worked again. I also thought about miniforge, but I didn’t want to :).

[gernophil]

Since your using slurm, I assume your working on an HPC. Somehow your folder structure is a bit weird. Looks like conda is installed directly in /home/usr. Is /home/usr a symlink to your actual home directory? Otherwise it looks to me like miniforge is installed somehow globally for all users. Don‘t you install it manually on your user folder?