Paralellize samtools piped with awk

Question

[Korsocius]

Dear all,

I have question. I would like to parallelize one line command. Could you help me please?

ls 4*.bam | parallel --pipe -j 4 -k samtools view -h {} |  awk '(length($10) < 180 || $1 ~/^@/)' | awk '(length($10)>35 || $1 ~/^@/)' | samtools view -bS -o  '5{}'

Is it possible to do that like this or I have to separate samtools and awk? Thank you

Answer 1

[michael.ante]

You can write a shell script doing what your pipe is doing and using parallel with that script.

[Kevin Blighe]

An expansion to Michael's script idea, you should do something like this:

ls 4*.bam | while read BAM
do
   [parallelised code executed on "${BAM}"]
done

Alternatively, you could split the output of ls into chunks and then run each in separate terminals:

ls 4*.bam | split --lines=4

This will create file 'chunks' called xaa xab xac, etc. You can run the same loop above (less parallel) multiple times in multiple sessions, and in this way it's parallelising in a different sense.