any idea why hisat2 is usurping all my RAM and threads

Question

[shripathiacademics]

Hi, I am trying to run hisat2 with 400 samples in a loop. for some reason when i specify -p it simply takes over all the resources in my computer and exits with error message "(ERR): hisat2-align exited with value 137". It works perfectly ok when don't specify any -p. commands i used

#!/bin/bash
cat /mnt/samplesNewList.txt  | while read line
do
sudo hisat2 -p 2  -x /mnt/genomeindex-hisat/genome  -U /mnt/$line.clean.fq.gz  -S $line.sam  --summary-file mnt/summary.txt &
done

any one encountered same issue?

[WouterDeCoster]

You should not use sudo for things like this, only for sysadmin stuff or installing new software globally.

Answer 1

[ATpoint]

Adding on h.mon, you also did not feed the stream from stdin to the loop. It should be

cat (...) | (...)
done < /dev/stdin

Still, I suggest using GNU parallel here. This allows to efficiently use your ressources while aligning several samples at once. Say you have 16 cores and want to run 4 samples in parallel:

cat /mnt/samplesNewList.txt | \
parallel -j 4 "hisat2 -p 4  -x $GENOME  -U /mnt/{}.clean.fq.gz  -S {}.sam  --summary-file mnt/{}.summary.txt"

Edit: To save disk space, consider to stream the output directly into samtools view to get a BAM file. Within the parallel command:

alignment | samtools view -bo {}.bam

[shripathiacademics]

Thank you. It worked.

Answer 2

[h.mon]

Because you are sending the hisat runs to the background (the & at the end, just before done), so the loops continues immediately to next iteration. This means you are aligning 400 samples simultaneously.

[WouterDeCoster]

To just run the alignments one after each other, remove the &.

To do some parallel work with more control, look at gnu parallel.

[shripathiacademics]

Thanks for the tip.