Snakemake >= version 8 cluster configuration and SLURM job submission

Question

[Maxwell]

I am trying to use snakemake to submit jobs onto my SLURM hpc, but since version 8 was released, snakemake no longer supports --cluster option.

Their documentation for how to conduct this workflow is quite confusing for version >= 8. With the latest version (8.9.1) I would like some assistance on how to run this command

snakemake -s /some/path/to/snakefile --cluster 'sbatch --time=48:00:00 --ntasks=1 --mem=100gb --nodes=1 --account=open' -j 8 -c 1

I tried this, but was unsuccessful: TypeError: SpawnedJobArgsFactory.general_args() got an unexpected keyword argument 'executor_common_settings'

snakemake -s /some/path/to/snakefile --executor cluster-generic --cluster-generic-submit-cmd 'sbatch --time=48:00:00 --ntasks=1 --mem=100gb --nodes=1 --account=open' -j 8 -c 1

I found this link, but im not really sure how this works either, its a bit unintuitive and I thought since snakemake is more of a bioinformatics tool, this post should be on biostars:

https://stackoverflow.com/questions/77929511/how-to-run-snakemake-8-on-a-slurm-cluster

Does anyone have any experience with this and can help? Thank you ahead of time and apologies if I did something wrong in post :)

[GenoMax]

As shown in the answer from the stackoverflow a new plugin needs to be installed using pip for snakemake 8 to work with clusters. Send the stackoverflow answer link to your cluster admins and ask them to install the plug-in. It may be best to have them do the install cluster-wide so others will benefit down the road.

After that is done your second command should start working.

[Maxwell]

oh i mean i have the plug-in installed. This error above still is generated so im not sure thats the problem, but i am clearly not doing something else correctly, like for example profiles or configuration? im not sure.

[GenoMax]

Did you install the plug-in yourself or the cluster admins did? If you installed it in your own account then it may not be available cluster-wide to SLURM. Also the -j 8 -c 1 should be inside quotes since they are SLURM options?

[Maxwell]

hmm I see okay, then I will reach out to my cluster admins and ask them to install. The -j 8 -c 1 are fine I believe. Thats how it is in the documentation and in the old version. (btw version 7 works fine, but I think its better to upgrade) Thank you for your help, I will def reach out.

[Mensur Dlakic]

Why isn't it easier to go back to Snakemake < 8 and not deal with these problems?

By the way, this is not a bioinformatics question.

[Maxwell]

Thank you for your comment Mensur Dlakic.

As listed in my OP, I felt this is appropriate for Biostars given that snakemake is primarily used in bioinformatics. If you believe it is not a bioinformatics question, Im curious to hear what your definition of a bioinformatics question is and why my definition is invalid. You should also then have an issue with all of the other workflow questions on biostars (including other less bioinformatics-y workflows besides Snakemake specific questions). I would also be interested to hear in some helpful advice on where to better post this question?

The question was not about going back to Snakemake < 8. The question was about transitioning into Snakemake > 8 given that it is almost Snakemake v9 and there are many benefits of using Snakemake > v8.

By the way, this is not a helpful answer and not an appropriate way of obtaining stars/experience.

Thank you any actual help or advice you may consider giving.

[Mensur Dlakic]

Bioinformatics questions are about bioinformatics. Your question is about a workflow and cluster maintenance, and it is more appropriate for forums like Stack Overflow. There isn't even a hint that you are using this to run bioinformatics programs in your command or how you asked the question. You can be indignant all you want, but this simply isn't a bioinformatics question. Lucky for you, there are kind and helpful moderators in this forum that will answer non-bioinformatics questions, even though that is applied in a non-consistent way.

I did not answer your question - I offered a comment instead. That's what happens when we post in a public forum - sometimes we have to put up with opinions we don't like.

In my opinion it is best not to upgrade anything unless there is a good reason such as the old program or package not working properly. Why? Because it often happens that a new package is not backwards compatible, which you are proving with your question. You don't have to like that opinion or may consider it unhelpful. I know that there wasn't anything offensive or inappropriate in my comment.

[Maxwell]

Do you have any resources or comments that will be helpful in accomplishing my original post or....? Sorry im confused why you are commenting? Can you help me or are you here to troll users?

Are you actually doing bioinformatics or are you here to tell people what bioinformatics is? Or should we also sit down and have a chat about public forums, or actually maybe you can educate me about the origin and utility of the internet?

Hey I have a great idea, you should also create an account on stackoverflow and mathoverflow and comment on everyones posts to tell them whether or not their post should be on stackoverflow/mathoverflow or whether it should just not be on the internet, or that its a bioinformatics question. Youd probably do well to explain to them the entire field of biology and what it is. Looking forward to your successful job well done on becoming the new internet moderator

Answer 1

[Simon]

Hello I was having the same problem, I followed the instruction to install the development version via pip

• installation of the plugin pip install snakemake-executor-plugin-slurm I am not getting the error message (still not running in my case but might be helpful) Best