Snakemake error: "Removing output files of failed job since they might be corrupted:"

Question

[kamanovae]

Hi!

I have a pipeline to process sequencing results, but the last rule gets an error:

Error in rule bwa_unmapped_hg38_samtools3:
    jobid: 8
    output: out/id_SRR13510812_unmapped_hg_2.txt
    shell:
        diff out/id_SRR13510812_2.txt out/id.SRR13510812.txt > out/id_SRR13510812_unmapped_hg_2.txt
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Removing output files of failed job bwa_unmapped_hg38_samtools3 since they might be corrupted:
out/id_SRR13510812_unmapped_hg_2.txt
[Thu Jun 24 14:18:52 2021]
Error in rule bwa_unmapped_hg38_samtools3:
    jobid: 7
    output: out/id_SRR13510812_unmapped_hg_1.txt
    shell:
        diff out/id_SRR13510812_1.txt out/id.SRR13510812.txt > out/id_SRR13510812_unmapped_hg_1.txt
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Removing output files of failed job bwa_unmapped_hg38_samtools3 since they might be corrupted:
out/id_SRR13510812_unmapped_hg_1.txt
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message

The last rule looks like this. I am using the diff line command to compare two samples:

rule bwa_unmapped_hg38_samtools3:
        input: data1=config["out"] +"id_{sample}_{read}.txt" , data2=config["out"] + "id.{sample}.txt"
        output: config["out"]+ "id_{sample}_unmapped_hg_{read}.txt"
        shell: "diff {input.data1} {input.data2} > {output}"

[liorglic]

There must be some other error message that you don't catch for some reason. I suggest you just copy one of the commands and run it manually and see what happens. Since you are using relative paths, I suspect that you'll see some "file does not exist" error, in which case you'll probably want to add some cd at the beginning of your shell command, or use full paths.

[kamanovae]

Thank you for your answer!

When I run the command outside of the snakemake, everything works and I get the required output files. I think the point is in the snakemake. I have no problem with paths for sure

Answer 1

[dariober]

diff returns exit code 1 if the files differ and that makes snakemake throw the error you see. Similar behaviour happens if grep doesn't find the search pattern in input, it returns non-zero exit code.

To make diff always return 0 you could do diff a.txt b.txt || true but that will ignore any error.

[kamanovae]

Thank you! But when I run just such a command, the result is not written to the output file, but displayed on the screen. I do not understand why

rule bwa_unmapped_hg38_samtools3:
        input: data1=config["out"] +"id_{sample}_{read}.txt" , data2=config["out"] + "id.{sample}.txt"
        output: config["out"]+ "id_{sample}_unmapped_hg_{read}.txt"
        shell: "diff {input.data1} {input.data2} || true > {output}"

[dariober]

Put "|| true" after "> {output}". Like diff a b > c || true