Can BWA restart a calculation after a break?

Question

[DNAngel]

I'm trying to streamline my analyses using bwa and samtools on different servers but I am only allowed to run big processes for 24 hours before it stops processes. The way around this is to have scripts/programs pick up from where it left off (like rsync can do). Is there a way to make bwa and samtools restart calculations without having to completely redo a whole session?

Thanks!

[Ram]

That is such a strange policy! I've heard of prohibiting long-running processes on login nodes, but clusters are meant to handle jobs that take a long time to get to completion. Maybe try talking to your sysadmin to check why processes are being restricted to 24h? Maybe they need special approval for longer processes.

[Pierre Lindenbaum]

same problem here :-) (see dariober's solution )

The joy of working with a cluster that limits the duration of the jobs to 24H00. pic.twitter.com/uObENFv2Dz

— Pierre Lindenbaum (@yokofakun) December 19, 2016

[GenoMax]

Sadly many (?) NGS programs are not check-point capable (some exceptions, e.g. most 10x genomics software). So if a job is interrupted you will have to start over.

• While not ideal you could try breaking your input files into multiple chunks and start multiple processes in parallel. Even if there is a limit on that it is hopefully reasonable like 10 or 20.
• You are also using multiple cores (both bwa and samtools will support it) for your jobs, right? That generally speeds things up.

[ATpoint]

If you share your code we can see if there are ways to optimize it in order to increase the speed. 24 hours is pretty long for alignment unless you are doing really large WGS or Hi-C datasets.

[DNAngel]

BWA for one species generally isn't that long, it's that my script will automatically run it for 30+ species. The samtools sort however can take a freakin long time and I have no idea how to accommodate that.

[GenoMax]

You can use multiple cores for samtools sort as well. You have not given us any information about how you are running these jobs.

Answer 1

[dariober]

A dumb solution may be to split the input fastq files in chunks small enough to fit the 24h limit, align each file-chunk and then merge. For file splitting, unix comes with the handy split command that you can use e.g. as:

zcat reads.R1.fq.gz | split -l 40000000 - reads.R1.fq.split
zcat reads.R2.fq.gz | split -l 40000000 - reads.R2.fq.split

this will split each input file in chunks of 40M lines (10M reads).

[DNAngel]

The alignment itself isn't long when I run it for one species - it's the samtools sort that takes a while too. But, my bash script will run the alignment for 30+ species which in total can take 2 days almost! I think I have to just add some pauses in the script which will somehow allow me to continue. But the dang samtools sort is just so slow!

[Pierre Lindenbaum]

But, my bash script will run the alignment for 30+ species which in total can take 2 days almost!

use a workflow manager (make, snakemake, etc..) . If the workflow manager itself takes more than 24H00, break your workflow into parts, the last action of workflow01 will call your cluster manager for workflow02 for the next 24H00, the last action of workflow02 will call your cluster manager for workflow03 for the next 24H00, etc...