Piping Bwa And Samtools

Question

[2184687-1231-83-]

Hi,

Can someone tell me how to pipe bwa and samtools to get to a pileup2fq from a "bwa sampe"? I am trying this but it uses a humongous amount of memory and doesn't produce any results past sampe:

~/src/bwa/latest/bwa-0.5.7/bwa sampe collapsed.fasta.gz collapsed.aln_sa1.sai collapsed.aln_sa2.sai *_?_1_pf.fastq *_2_pf.fastq |\
~/src/samtools/latest/samtools/samtools view -bt collapsed.fasta.gz - |\
 ~/src/samtools/latest/samtools/samtools sort -m 9000000 - - |\
 ~/src/samtools/latest/samtools/samtools pileup -cf collapsed.fasta.gz - |\
 perl ~/src/samtools/latest/samtools/misc/samtools.pl pileup2fq -D1 |\
 gzip > collapsed.final.fastq.gz

[Jeremy Leipzig]

I would be a little cautious about using wildcards with BWA. I tried that once and it took the first file and ignored the rest.

[Pierre Lindenbaum]

nice pipeline. It's a good summary of the bwa/pileup workflow.

Answer 1

[Istvan Albert]

When you pipe the programs all of them consume memory at the same time. Probably your best solution is to direct the output into temporary files.

Answer 2

[Pablo Marin-Garcia]

Not answering your qustion but.. Is it really a good idea to pipe to samtools sort from bwa?

Sorting is very memory expensive and as a general rule if the previous step is taking long time to finish and produce big quantity of data you end having the program doing the sort waiting for long time sitting idle and consuming a lot of memory, and if you are working in parallel in a multi-core machine, usually memory is the bottleneck. Would it be this the case of bwa | samtools sort? or they behave differently?