snakemake job status monitoring

Question

[Eugene A]

Hi everyone, I'm writing the simple linear snakemake pipeline on multiple samples.

I'm runnign snakemake with -k parameters, so failing in one branch does not cause to fail whole pipeline. But I'd like also to have a meaningfull info (possibly via emails) on the pipeline status (which joabs are done, and which are failed). There is a --log-handler-script parameter which supposetly should give me necessary functioonality. BUT, the msg's produced by sankemake rather uninformative:

as one might see the one of the job failed but msg (i just printed it to terminal from the def log_handler(msg):) does not tell me which step went wrong. And I do not see how I can get the jobID of failed job before I the pipeline is over. Any suggestions are appreciated

Best, Eugene

[Jeremy Leipzig]

if something crashed it is usually pretty easy to figure out what happened but if a task exited with a normal status code but a file is missing then yes it won't be super helpful

[Eugene A]

It is pritty easy to figure out what happend in manual examination. My task was to automatically detect failed rules and report them (via emails or something like that), while the rest of a pipeline goes on (-k option). And _automatic_ detection of failed jobs is surprisengly inconvenientin snakemake

[lianov]

I assume this something you may have already added, but to be sure, do you have the log directives as part of your rules? https://snakemake.readthedocs.io/en/stable/tutorial/advanced.html?highlight=log%20directive#step-5-logging

This allows you to see errors more specifically linked to the jobs submitted.

As for emails I find the following helpful: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html?highlight=onstart#onstart-onsuccess-and-onerror-handlers

Answer 1

[Eugene A]

after some more search I've figured that there is other option for monitoring appart from --log-handler-script, namely panoptes server (https://github.com/panoptes-organization/panoptes, https://snakemake.readthedocs.io/en/stable/executing/monitoring.html). It also has some problems (failed jobs currently displayed as running, so does not solve initial issue), but will suite my purpuses for now.

Answer 2

[Eugene A]

Some additions, which might be helpfull for someone - it seems that the only way to detect failed jobs in snakemake and automatically keep track of them is to intercept exit codes of the used commands and processe it separetly