Snakemake-Aligment using BWA-MEM2

Question

[priya.bmg]

Hello

I have started using snakemake 6.5.2 to align fastq files with reference file. I have pasted the error below in this question. How to allocate memory in the snakefile and read the header from samfile, '-'. This is the snakefile (wrapper for running alignment):

  rule bwa_mem2_mem:
        input:
            reads=["/scicore/home/cichon/GROUP/test_workflow/samples/{sample}.1.fq", "/scicore/home/cichon/GROUP/test_workflow/samples/{sample}.2.fq"]
        output:
            "/scicore/home/cichon/GROUP/test_workflow/output/{sample}.bam"
        log:
            "/scicore/home/cichon/GROUP/test_workflow/output/{sample}.log"
        params:
            index="/scicore/home/cichon/GROUP/test_workflow/data/gch38.fa",
            extra=r"-R '@RG\tID:{sample}\tSM:{sample}'",
            sort="none",             # Can be 'none', 'samtools' or 'picard'.
            sort_order="coordinate", # Can be 'coordinate' (default) or 'queryname'.
            sort_extra=""     

        threads: 8
        wrapper:
            "0.77.0/bio/bwa-mem2/mem"

I used this command to execute the above snakemake file:

snakemake --use-conda --snakefile Snakefile --cores -p /scicore/home/cichon/GROUP/test_workflow/output/DE98NGSUKBD117612_1.bam /scicore/home/cichon/GROUP/test_workflow/output/DE71NGSUKBD117613_3.bam

Error from log file:

Looking to launch executable "/scicore/home/cichon/GROUP/test_workflow/.snakemake/conda/0ecf6ba401c70d7501c73bb1c733a060/bin/bwa-mem2.avx", simd = .avx
Launching executable "/scicore/home/cichon/GROUP/test_workflow/.snakemake/conda/0ecf6ba401c70d7501c73bb1c733a060/bin/bwa-mem2.avx"
-----------------------------
Executing in AVX mode!!
-----------------------------
* SA compression enabled with xfactor: 8
* Ref file: /scicore/home/cichon/GROUP/test_workflow/data/gch38.fa
* Entering FMI_search
* Index file found. Loading index from /scicore/home/cichon/GROUP/test_workflow/data/gch38.fa.bwt.2bit.64
* Reference seq len for bi-index = 6544178411
* sentinel-index: 2782712944
* Count:
0,  1
1,  1914743810
2,  3272089206
3,  4629434602
4,  6544178411

* Reading other elements of the index from files /scicore/home/cichon/GROUP/test_workflow/data/gch38.fa
* Index prefix: /scicore/home/cichon/GROUP/test_workflow/data/gch38.fa
* Read 0 ALT contigs
[fread] Cannot allocate memory
[main_samview] fail to read the header from "-".

Thanks

[ATpoint]

How much memory is available /allocated on that machine you run it on?

[priya.bmg]

1 GB of memory for each core. 20 cores will be used by default

[ATpoint]

But here you used 8 cores, is it? So 8GB, that is not enough. See the bwa-mem2 GitHub page, the human index on disk alone is 10GB.

[priya.bmg]

I have specified number of threads as 8 and given the number of cores as default (which is 20)

[ATpoint]

You are out of memory apparently, so try to find out why. Run the command outside of the snakefile and see whether it works, e.g. on a small subset of reads. That'll tell you whether the machine itself can load the index into memory.

[priya.bmg]

Earlier, I have used 8 cores with 10GB of memory allocated to each. In Snakemake, where do I specify the memory?

[Hyper_Odin]

https://stackoverflow.com/questions/60403990/how-to-set-binding-memory-limits-in-snakemake

[priya.bmg]

Not sure. When I include memory limit in rule file, it is shown as syntax error.

[Hyper_Odin]

Can you post the error? How did you call snakemake after including the rule?

Here is the complete snakemake documentation of the above thread:

Skip to the topic "Resources"

[priya.bmg]

Sorry, I included mem_gb in the rule and it showed syntax error. Now, I changed to mem_mb and I get attribute error. Is there any error in the wrapper script?

snakemake --resources mem_mb=100 #command used to run snake make file

 snakemake --resources mem_mb=100
Traceback (most recent call last):
  File "/scicore/home/cichon/thirun0000/miniconda3/envs/snakemake_6.5.2/bin/snakemake", line 10, in <module>
    sys.exit(main())
  File "/scicore/home/cichon/thirun0000/miniconda3/envs/snakemake_6.5.2/lib/python3.9/site-packages/snakemake/__init__.py", line 2400, in main
    resources = parse_resources(args.resources)
  File "/scicore/home/cichon/thirun0000/miniconda3/envs/snakemake_6.5.2/lib/python3.9/site-packages/snakemake/resources.py", line 85, in parse_resources
    for res, val in resources_args.items():
AttributeError: 'list' object has no attribute 'items'

[Hyper_Odin]

I think you should update your snakemake to the latest version. Trouble running snakemake while defining maximum RAM usage

EDIT: Or see if downgrading helps. Downgrade to 6.4.1