Snakemake issue with wrappers

Question

[Begonia_pavonina]

I have issues when running a wrapper of BWA mem with Snakemake. The error message "No module named 'snakemake_wrapper_utils'" appear (see below).

However, when checking if the package is installed in Python, I found the following:

import snakemake_wrapper_utils
print(snakemake_wrapper_utils.__version__)
0.1.0

Did anyone have this problem? Would you know why there is a module error, while the module is installed?

snakemake --cores all

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Conda environments: ignored
Job stats:
job        count
-------  -------
all            1
bwa_mem        3
total          4

Select jobs to execute...

[Thu Nov  2 17:43:16 2023]
rule bwa_mem:
    input: reads/110627_0240_AC0254ABXX_2_SA-PE-001.1.fastq.gz, reads/110627_0240_AC0254ABXX_2_SA-PE-001.2.fastq.gz, genome.amb, genome.ann, genome.bwt, genome.pac, genome.sa
    output: mapped/110627_0240_AC0254ABXX_2_SA-PE-001.bam
    log: logs/bwa_mem/110627_0240_AC0254ABXX_2_SA-PE-001.log
    jobid: 1
    reason: Missing output files: mapped/110627_0240_AC0254ABXX_2_SA-PE-001.bam
    wildcards: sample=110627_0240_AC0254ABXX_2_SA-PE-001
    resources: tmpdir=/tmp/tmichel_8894483

Traceback (most recent call last):
  File "/mnt/shared/scratch/usr/comp_baits_pseudo-chr/baits/.snakemake/scripts/tmpya0pk1fa.wrapper.py", line 14, in <module>
    from snakemake_wrapper_utils.java import get_java_opts
ModuleNotFoundError: No module named 'snakemake_wrapper_utils'
[Thu Nov  2 17:43:21 2023]
Error in rule bwa_mem:
    jobid: 1
    input: reads/110627_0240_AC0254ABXX_2_SA-PE-001.1.fastq.gz, reads/110627_0240_AC0254ABXX_2_SA-PE-001.2.fastq.gz, genome.amb, genome.ann, genome.bwt, genome.pac, genome.sa
    output: mapped/110627_0240_AC0254ABXX_2_SA-PE-001.bam
    log: logs/bwa_mem/110627_0240_AC0254ABXX_2_SA-PE-001.log (check log file(s) for error details)
    conda-env: /mnt/shared/scratch/usr/comp_baits_pseudo-chr/baits/.snakemake/conda/8ec8a2e6cccaa67a3428f1bf7dcc48b8_

RuleException:
CalledProcessError in file /mnt/shared/scratch/usr/comp_baits_pseudo-chr/baits/workflow/Snakefile, line 76:
Command 'set -euo pipefail;  /mnt/shared/home/usr/mambaforge/envs/snakemake/bin/python3.12 /mnt/shared/scratch/usr/comp_baits_pseudo-chr/baits/.snakemake/scripts/tmpya0pk1fa.wrapper.py' returned non-zero exit status 1.
  File "/mnt/shared/scratch/usr/comp_baits_pseudo-chr/baits/workflow/Snakefile", line 76, in __rule_bwa_mem
  File "/mnt/shared/home/usr/mambaforge/envs/snakemake/lib/python3.12/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2023-11-02T174251.293416.snakemake.log

[Eric Lim]

I noticed conda-env in the error message. Wrapper defines its own environment to run the wrapper, I'd remove the conda: attribute from your rule and try again with snakemake --use-conda. Hopefully that fixes the issue.

[Begonia_pavonina]

Thank you Eric Lim, I had a conda activate environement in the batch script I was using to run Snakemake. Unfortunately, removing it does not change the previous error.

import os
import snakemake.io
import glob
import pandas as pd

configfile: "config/config.yaml"

table=pd.read_csv("config/table_reads.tsv", delim_whitespace=True,  header=None, index_col=False)

SAMPLES=table.loc[:, 0].to_list()
READS=["1","2"]

rule all:
        input:
                expand("mapped/{sample}.bam", sample=SAMPLES)

rule fastqc:
        input:
                expand("reads/{sample}.{read}.fastq.gz", sample=SAMPLES, read=READS)


        output:
                html="qc/fastqc/{sample}.html",
                zip="qc/fastqc/{sample}_fastqc.zip" # the suffix _fastqc.zip is necessary for multiqc to find the file. If not using multiqc, you are free to choose an arbitrary filename
        params:
                extra = "--quiet"
        log:
                "logs/fastqc/{sample}.log"
        threads: 1
        resources:
                mem_mb = 1024
        wrapper:
                "v2.6.0/bio/fastqc"

rule bwa_index:
        input:
                config["reference"]
                #"{genome}.fasta", 
        output:
                idx=multiext("{genome}", ".amb", ".ann", ".bwt", ".pac", ".sa"),
        log:
                "logs/bwa_index/{genome}.log",
        params:
                algorithm="bwtsw",
        wrapper:
                "v2.6.0/bio/bwa/index"

rule bwa_mem:
        input:
                reads=["reads/{sample}.1.fastq.gz", "reads/{sample}.2.fastq.gz"],
                idx=multiext("genome", ".amb", ".ann", ".bwt", ".pac", ".sa"),
        output:
                "mapped/{sample}.bam",
        log:
                "logs/bwa_mem/{sample}.log",
        params:
                extra=r"-R '@RG\tID:{sample}\tSM:{sample}'",
                sorting="none",  # Can be 'none', 'samtools' or 'picard'.
                sort_order="queryname",  # Can be 'queryname' or 'coordinate'.
                sort_extra="",  # Extra args for samtools/picard.
        threads: 8
        wrapper:
                "v2.9.0-45-g0055391/bio/bwa/mem"

[Begonia_pavonina]

I have noticed that adding "import snakemake_wrapper_utils" in the Snakefile trigger the error:

No module named 'snakemake_wrapper_utils'

So there is definitely a problem with this module, which I can find in Python, but not running Snakemake.

[Ram]

Can you print sys.path both in python and within the snakemake script and see if the results are the same?

[Begonia_pavonina]

Thank you Ram, I am not sure if I should have the same result.

In Snakemake:

print(sys.path)
['/mnt/shared/scratch/usr/comp_baits_pseudo-chr/baits/workflow', '/mnt/shared/home/usr/mambaforge/envs/snakemake/bin', '/mnt/shared/home/usr/mambaforge/envs/snakemake/lib/python312.zip', '/mnt/shared/home/usr/mambaforge/envs/snakemake/lib/python3.12', '/mnt/shared/home/usr/mambaforge/envs/snakemake/lib/python3.12/lib-dynload', '/mnt/shared/home/usr/mambaforge/envs/snakemake/lib/python3.12/site-packages', '/opt/gurobi201/linux32/lib/python2.5']

In python:

>>> print(sys.path)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'sys' is not defined

[Ram]

You should really have done this on your own, but import sys then print(sys.path) to address the error in the second case. The results should at least overlap.

[Begonia_pavonina]

Ah, I did not know sys was a Python library. It is worth mentioning it.

print(sys.path)
['/mnt/shared/scratch/usr/comp_baits_pseudo-chr/baits/workflow', '/mnt/shared/home/usr/mambaforge/envs/snakemake/bin', '/mnt/shared/home/usr/mambaforge/envs/snakemake/lib/python312.zip', '/mnt/shared/home/usr/mambaforge/envs/snakemake/lib/python3.12', '/mnt/shared/home/usr/mambaforge/envs/snakemake/lib/python3.12/lib-dynload', '/mnt/shared/home/usr/mambaforge/envs/snakemake/lib/python3.12/site-packages', '/opt/gurobi201/linux32/lib/python2.5']

>>> import sys
>>> print(sys.path)
['', '/mnt/shared/home/tmichel/mambaforge/envs/snakemake/lib/python312.zip', '/mnt/shared/home/tmichel/mambaforge/envs/snakemake/lib/python3.12', '/mnt/shared/home/tmichel/mambaforge/envs/snakemake/lib/python3.12/lib-dynload', '/mnt/shared/home/tmichel/mambaforge/envs/snakemake/lib/python3.12/site-packages']

[Ram]

I'm going to make a wild guess here. The snakemake env you have under the tmichel home dir is different from the snakemake env you have with the /mnt/shared/ prefix. The latter is missing the required library.

[Begonia_pavonina]

I think the two different environment are at the source of the problem indeed.

I am not sure it is due to a missing snakemake_wrapper_utils module, as I have installed it in both environment. But it might be related to the Python version loaded.

There is a python2.5 in the snakemake environment, and I think it might cause the snakemake_wrapper_utils module not to be found. I am trying to find a way not to load python2.5.