What are fastest possible ways to sort the large bam files (greater then 100GB) (and With new version of highseq's; data will further increase in size...)
samtools sort -m 6000000000
Picard's SortSam
Any others tools or multicore/parallel versions? and best possible options?
How are large sequencing centers dealing with this problem
A utility from novocraft is the best, but its license expires in 15 days, if I remember correctly. There is a multithreaded sort I implemented as a weekend project (in the "mt" branch), but it is not as efficient because fully parallelizing needs a reimplementation, while I only have the time for small modifications. With the samtools-mt, you can:
samtools sort -@ 8 -m 4G
Note that each thread will use about 4-4.5GB of memory in this setting, so make sure you have enough memory when pushing up -m and -@.
Sorting a 100GB BAM takes a little more than 1 day as I remember. Multithreaded samtools saves you about half a day with 4-8 cores. Novocraft is even better.
I know I'm late to the party but I just tried this tool: http://lomereiter.github.io/sambamba/
and it sorted by bam 4 times faster than samtools sort.
Hi,
The latest Novosort can also Mark/Remove duplicates while sorting. This adds negligible time to the sort so effectively you get mark duplicates for free. Marking is similar to Picard process.
Best, Colin
One more tool to consider if the other ones do not give you satisfaction: biobambam. Disclaimer: I have not tested it yet.
I think i saw on the samtools listserv someone saying the best way would be to split up the bam into uncompressed bams, sort it and then merge it back together. That way the sorting would be done in parallel.
Edit: actually nevermind, see here: https://sourceforge.net/mailarchive/message.php?msg_id=28795254
Hi, Latest Novosort has trial license until 1st September, after that it's $299/yr for license. Latest Version has:
Cheers, Colin
Latest Novosort can also Mark/Remove duplicates while sorting at virtually no performance loss compared to just sorting.
Duplicate detection also allows use of molecular bar code tags in the signature.
One more option by DNANexus that tries to obtain a speedup by using a RocksDB backend (trying to improve file I/O from what I can gather) http://devblog.dnanexus.com/faster-bam-sorting-with-samtools-and-rocksdb/ uses a fork of samtools on github
Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
version 2.3.6
I may be wrong: novosort seems to be binary free...
I actually I was correct on the novosort licensing. When the license expires, mutli-threading will be switched off, though its single-threaded version is faster than samtools' single-threaded sort.
I believe that nowadays the "mt" branch of samtools is now merged into the trunk at github. I post this comment to warn other readers because I spend quite some time unsuccesfully trying to find the "mt" branch.
I mean the free license expires in 15 days, if I am correct.
thanks @lh3: I need to test novosoft on multiple threads and also need to see if sorted bam is compatible with GATK and other VC tools
I actually I was correct on the novosort licensing. When the license expires, you will be left with a single-threaded version. I do not know how much novosort costs, but it is really great.