Running python script in slurm using conda environment

Question

[sidrah.maryam]

Hello everyone! I am trying to run slurm using python in the environment, but the base is overwriting it and due to which the tool is not running.

#!/bin/bash
#SBATCH --ntasks-per-node=1
#SBATCH --mem-per-cpu=100G
#SBATCH --time=100:00:00
#SBATCH --job-name=celloracle
#SBATCH --output=celloracle-%J.log
# Load the same python as used for installation
#module load python3
module load R/4.1.2
source /home/user/env/lib/python3.7
#source activate evn_for_R
python celloracle_issue.py

however, it is still not working. Has anyone ever encountered same situation? Any help is highly appreciated.

Answer 1

[ATpoint]

I used this snipped for years with slurm:

eval "$(conda shell.bash hook)"
conda activate your_env

I encourage to use everything via sconda or a container rather than modules. Modules can change over time, making them (strictly speaking) not reproducible.

Answer 2

[bk11]

You need to active your conda environment first-

source /PathToYourCondaPackage/bin/activate

and check

which python
/PathToYourCondaPackage/bin/python

Answer 3

[Barry Digby]

You can achieve this using conda run. Assuming the name of your conda environment is my_env:

#!/bin/bash
#SBATCH --ntasks-per-node=1
#SBATCH --mem-per-cpu=100G
#SBATCH --time=100:00:00
#SBATCH --job-name=celloracle
#SBATCH --output=celloracle-%J.log

conda run -n my_env python celloracle_issue.py

Make sure you have the latest conda version conda update -n base conda

Answer 4

[filip.buric]

While not addressing your problem directly, I highly recommend wrapping your conda environment in a Singularity container. This has many advantages, mainly:

• speed (Python envs are comprised of many small module files and simply loading those can introduce an IO bottleneck)
• ease of management (you just use a single container file which you run with a single command, as opposed to tens or hundreds of thousand of files for a single conda env, which can quickly take a large chunk of your file quota)
• and the fact that the environment is now "frozen" (you can't modify it by mistake and it's more reproducible).

I wrote a quick guide here: https://github.com/fburic/notes/blob/master/singularity_conda.md (some parts of the Singularity recipe are specific to my old uni cluster, but yours should have examples as well, so should be easy to adapt).

If you haven't used Singularity before, the basic idea is rather simple: All your software dependencies (not limited to Python packages) are stored inside a single file, which you can think of as a "virtual machine" (well, a lightweight one, but just for the intuition).

Using it is very simple, just: singularity exec my_container.sif python my_script.py