Trouble running snakemake while defining maximum RAM usage

Question

[cfos4698]

Hi,

One rule in my snakemake workflow (in which I run kraken2) has a large RAM requirement of ~50 GB. When running the workflow on many samples in parallel (e.g., snakemake --cores 12 -s workflow.smk), I get a bunch of errors because the system's RAM is exhausted.

As a result, I've tried limiting the amount of RAM used by snakemake by setting a maximum memory usage within the kraken2 rule:

    resources:
        mem_mb=55000

Intuitively, I thought that this would mean that snakemake would run each job at a time since running >1 job would exceed 55000 MB (=55 GB) of RAM usage. However, the workflow still fails by running out of RAM despite my system having 64 GB of RAM.

Next, I tried also setting a maximum amount of RAM on the command line using the --resources flag:

snakemake -j12 -s workflow.smk --resources mem_mb=55000

However, I get a traceback error:

Traceback (most recent call last):
  File "/home/cfos/.local/bin/snakemake", line 8, in <module>
    sys.exit(main())
  File "/home/cfos/.local/lib/python3.8/site-packages/snakemake/__init__.py", line 2400, in main
    resources = parse_resources(args.resources)
  File "/home/cfos/.local/lib/python3.8/site-packages/snakemake/resources.py", line 85, in parse_resources
    for res, val in resources_args.items():
AttributeError: 'list' object has no attribute 'items'

What am I doing wrong here? Any advice would be appreciated.

[dariober]

Which verision of snakemake are you using? Also, try to write a minimal, self-contained example that reproduces the issue and post it here if you cannot find a solution.

[cfos4698]

Good point. I'm using snakemake v6.5.1 installed via mamba. Minimal example:

test.smk:

rule all:
    input:
        "test.txt"
    message:
        "This is a test!"

rule touch:
    output:
        file = "test.txt"
    resources: mem_mb=55000
    shell:
        """
        touch {output.file}
        """

Command:

snakemake -j1 -s test.smk --resources mem_mb=50000

Output:

Traceback (most recent call last):
  File "/home/vrl/miniconda3/bin/snakemake", line 10, in <module>
    sys.exit(main())
  File "/home/vrl/miniconda3/lib/python3.8/site-packages/snakemake/__init__.py", line 2400, in main
    resources = parse_resources(args.resources)
  File "/home/vrl/miniconda3/lib/python3.8/site-packages/snakemake/resources.py", line 85, in parse_resources
    for res, val in resources_args.items():
AttributeError: 'list' object has no attribute 'items'

Answer 1

[dariober]

UPDATE: This has been fixed in version 6.5.3

Answer 2

[dariober]

I think you found a bug in v6.5.1. I tested your example with 6.0.0 and it worked, upgraded to 6.5.1 and I can reproduce the error (don't know about versions in between).

EDIT

The issue has been submitted to github just few hours after you posted your question!

[cfos4698]

Just as an update, I upgraded to v6.5.2 and the issue still persists. I downgraded to 6.4.1 and the issues does not occur.

[cfos4698]

Excellent, good to know that I'm not going crazy! Thanks for pointing that out.