conda env in snakemake

Question

[Omics data mining]

Hello everyone

I am using snakemake to run in the HPC cluster. For each rule, I have conda file stored in "new_env.yaml" and running snakemake from base enviroment. For this, I have tried two different ways to run the script .

Option 1: Defined the path of conda enviroment configfile:"config.yaml"

rule seurat_individual: 
       input:
           config['sample']
       output:
          "TEST_out"
       conda: 
          "/home/.conda/envs/new_env"
       shell:
          "Rscript /home/scripts/Test.R {input}" 

Option 2: Defined the yaml file in conda

configfile:"config.yaml"

rule seurat_individual: 
       input:
           config['sample']
       output:
          "TEST_out"
       conda: 
          "/home/new_env.yaml"
       shell:
          "Rscript /home/scripts/Test.R {input}" 

Running option : snakemake --cluster sbatch -s Snakefile -j 1 --latency-wait 30

Both the options gives me error. Main error is snakemake script can not create "environment" and activates it to use the required packages to run the R script. How can I fix this issue. I would appreciate all the suggestion.

Thanks in advance

Answer 1

[colindaven]

This path might be incorrect:

  conda: 
      "/home/.conda/envs/new_env"    <----- try /home/SOMEUSER/.conda/envs  
   shell:
      "Rscript /home/scripts/Test.R {input}" 

It's good practice to check the paths in your scripts using ls before starting to avoid errors.

It might be something else though of course ...

Also this looks wrong too - I would expect /home/user/scripts "Rscript /home/scripts/Test.R {input}"

Also, if running on a cluster, it's better to place everything in cluster accessible directories

like this:

  /mnt/clusterstorage/user/scripts

[Omics data mining]

I did not post full detail before and have made changes in the "options" used

Option1 :

<h6>#</h6>

rule seurat_individual: 
       input:
           config['sample']
       output:
          "Test.object"
       conda: 
          "/home/user/.conda/envs/new_env"
       shell:
          "Rscript /work_ifs/user/SNAKEMAKE/ST/scripts/test.R {input}" 

Option2 :

<h6>#</h6>

rule seurat_individual: 
       input:
           config['sample']
       output:
          "TEST_out"
       conda: 
          "/home/user/path/new_env.yaml"
       shell:
          "Rscript /home/scripts/Test.R {input}" 

I have entered the exact path. but still it does not work.

[colindaven]

As following this tutorial Creating workflows with snakemake and conda

you could activate the conda before submission.

I much prefer nextflow to snakemake so if all your paths are ok - I don't really think this is the case - then I can't help much more. Conda is very well handled by both systems though.

[Omics data mining]

Its not possible to install all the methods in same conda environment as dependency packages are causing issue. This is why it was important for me to have "conda env" running in the script. But I will try to debug it and spend some time. Thanks alot