Is it possible to pass command line arguments to python script in snakemake?

Question

[blackadder]

Hello all,

When I am executing my snakemake pipeline I am trying to pass command line arguments with the --config flag in a python script.

rule assembly:
input:
    "trim/{sample}_1.fastq.gz",
    "trim/{sample}_2.fastq.gz",
output:
    "assembly/{sample}.fa"
params:
    fastq1 = "trim/{sample}_1.fastq.gz",
    fastq2 = "trim/{sample}_2.fastq.gz",
conda: "environment.yaml"
script:
    "assembly.py"

The Python script

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--config', type=str, required=True)
args = parser.parse_args()

if args.config["assembly"] == "megahit":
    shell ("megahit .........")
elif config["assembly"] == "metaspades":
    shell ("metaspades.py ........")

I am executing the pipeline like this:

snakemake --cores 5 --config assembly="megahit"

or

snakemake --cores 5 --config assembly="metaspades" 

When I do that I am getting the following error:

error: the following arguments are required: --config

1. Is there a way to make this work?
2. Am I completely wrong by choosing this option? (I did because when I used the snakemake run directive I was getting the "conda environments are only allowed with shell script notebook or wrapper directives (not with run)" error.

Thanking you in advance!

[Jeremy Leipzig]

i don't think you need to add the addparse stuff, just use a generic configfile with a default value and your inline arguments should work

[blackadder]

So, you suggest instead of --config use the --configfile and specify the different params in this one?

[Jeremy Leipzig]

all this is unnecessary:

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--config', type=str, required=True)
args = parser.parse_args()

[blackadder]

When i have the following code in the script :

if config["assembly"] == "megahit":
     shell ("megahit .........")
elif config["assembly"] == "metaspades":
     shell ("metaspades.py ........")

I am getting the following error:

if config["assembly"] == "megahit": NameError: name "config" is not defined

and I execute my pipeline like this

snakemake --cores 5 --config assembly="megahit"

[Jeremy Leipzig]

config won't exist unless you have a configfile. do you have one?

[cpad0112]

May be gb 's answer to your earlier post is useful to you: Snakemake command line arguments.

[blackadder]

By using the run directive, you cannot use conda environments... I need to load my programs from conda

[gb]

You could do something like (unstested and quickly typed so only for inspiration):

rule assembly:
input:
    "trim/{sample}_1.fastq.gz",
    "trim/{sample}_2.fastq.gz",
output:
    "assembly/{sample}.fa"
params:
    method = config["assembly"]
conda: "environment.yaml"
script:
    "assembly.py"

And in the script:

if snakemake.params.method == "megahit":
    shell ("megahit .........")
elif snakemake.params.method == "metaspades":
    shell ("metaspades.py ........")

But I personally prefer something you are already doing:

rule assembly:
input:
    "trim/{sample}_1.fastq.gz",
    "trim/{sample}_2.fastq.gz",
output:
    "assembly/{sample}.fa"
params:
    fastq1 = "trim/{sample}_1.fastq.gz",
    fastq2 = "trim/{sample}_2.fastq.gz",
conda: "environment.yaml"
script:
    'assembly.py --assembly config["assembly"]'

and

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--assembly', dest='assembly' type=str, required=True)
args = parser.parse_args()

if args.assembly== "megahit":
    #guess you want subprocess.Popen
    shell ("megahit .........")
elif args.assembly== "metaspades":
    shell ("metaspades.py ........")

The above is just to push you in the right direction. If you can't figure it out I can make a working example. Also the shell line in the python script looks weird but for now I assume it is just a example or not real code.

[blackadder]

Hello!

Thank you for the reply. Yes, the shell directive is an example and not the real code.

I will try your suggestion and I will get back to you.

Answer 1

[Eric Lim]

argparse is not needed.

(base) [~/Downloads/scratch/biostar/tmp]$ cat Snakefile 
rule:
    params:
        test = config['test']
    script:
        'test.py'

You can access config or params in your script with the snakemake object.

(base) [~/Downloads/scratch/biostar/tmp]$ cat test.py 
print(snakemake.config)

print(snakemake.params)

This works for config or configfile.

(base) [~/Downloads/scratch/biostar/tmp]$ snakemake -j1 --config test=hello
Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job      count    min threads    max threads
-----  -------  -------------  -------------
1            1              1              1
total        1              1              1

Select jobs to execute...

[Tue Jan 18 13:24:23 2022]
rule 1:
    jobid: 0
    resources: tmpdir=/var/folders/q4/cwjcyc851ldgbw2v28txzxbh0000gn/T

{'test': 'hello'}
hello
[Tue Jan 18 13:24:25 2022]
Finished job 0.
1 of 1 steps (100%) done

[Ecstasia]

I wouldn't agree that you never need to pass arguments to a python script in a snakemake workflow.

What if the python script you're trying to run is actually a published program and you just want to run it with certain parameters? Would I then have to run it from the shell directive?