Why sortmerna 4.2.0 runs so slow?

Question

[tianshenbio]

I was previously using 2.1.0 but it occasionally produced corrupted files under the pair-end mode, according to a previous post here I changed to the new version: 4.2.0.

I tested one pair of paired-end files (60M reads in total)

2.1.0 took ~1.5 hours to complete
4.2.0 took >12 hours to complete

How come it runs much slower?

The script I use for 2.1.0:

echo "Running SortMeRNA for $BASE"
/home/nus/Desktop/NGS/sortmerna-2.1b/sortmerna --ref /home/nus/Desktop/NGS/sortmerna-2.1b/rRNA_databases/silva-bac-16s-id90.fasta,/home/nus/Desktop/NGS/sortmerna-2.1b/index/silva-bac-16s-db:
/home/nus/Desktop/NGS/sortmerna-2.1b/rRNA_databases/silva-bac-23s-id98.fasta,/home/nus/Desktop/NGS/sortmerna-2.1b/index/silva-bac-23s-db:
/home/nus/Desktop/NGS/sortmerna-2.1b/rRNA_databases/silva-arc-16s-id95.fasta,/home/nus/Desktop/NGS/sortmerna-2.1b/index/silva-arc-16s-db:
/home/nus/Desktop/NGS/sortmerna-2.1b/rRNA_databases/silva-arc-23s-id98.fasta,/home/nus/Desktop/NGS/sortmerna-2.1b/index/silva-arc-23s-db:
/home/nus/Desktop/NGS/sortmerna-2.1b/rRNA_databases/silva-euk-18s-id95.fasta,/home/nus/Desktop/NGS/sortmerna-2.1b/index/silva-euk-18s-db:
/home/nus/Desktop/NGS/sortmerna-2.1b/rRNA_databases/silva-euk-28s-id98.fasta,/home/nus/Desktop/NGS/sortmerna-2.1b/index/silva-euk-28s:
/home/nus/Desktop/NGS/sortmerna-2.1b/rRNA_databases/rfam-5s-database-id98.fasta,/home/nus/Desktop/NGS/sortmerna-2.1b/index/rfam-5s-db:
/home/nus/Desktop/NGS/sortmerna-2.1b/rRNA_databases/rfam-5.8s-database-id98.fasta,/home/nus/Desktop/NGS/sortmerna-2.1b/index/rfam-5.8s-db 
--reads "${BASE}_merge.fq" 
--aligned "${BASE}_rRNA" 
--other "${BASE}_clean" 
--log -a 24 -v --paired_in --fastx

echo "Unmerging SortMeRNA filtered pairs for $BASE"
/home/nus/Desktop/NGS/sortmerna-2.1b/scripts/unmerge-paired-reads.sh "${BASE}_clean.fq" "${BASE}_clean1.fq" "${BASE}_clean2.fq"

The script I used for 4.2.0:

 rm /home/nus/sortmerna/run/kvdb/*
 echo "Running SortMeRNA for ${i}"
/home/nus/Desktop/NGS/sortmerna-4.2.0-Linux/bin/sortmerna 
-ref /home/nus/Desktop/NGS/sortmerna-2.1b/rRNA_databases/silva-bac-16s-id90.fasta 
-ref /home/nus/Desktop/NGS/sortmerna-2.1b/rRNA_databases/silva-bac-23s-id98.fasta 
-ref /home/nus/Desktop/NGS/sortmerna-2.1b/rRNA_databases/silva-arc-16s-id95.fasta 
-ref /home/nus/Desktop/NGS/sortmerna-2.1b/rRNA_databases/silva-arc-23s-id98.fasta 
-ref /home/nus/Desktop/NGS/sortmerna-2.1b/rRNA_databases/silva-euk-18s-id95.fasta 
-ref /home/nus/Desktop/NGS/sortmerna-2.1b/rRNA_databases/silva-euk-28s-id98.fasta 
-ref /home/nus/Desktop/NGS/sortmerna-2.1b/rRNA_databases/rfam-5s-database-id98.fasta 
-ref /home/nus/Desktop/NGS/sortmerna-2.1b/rRNA_databases/rfam-5.8s-database-id98.fasta 
-reads ${i}_1P.fq.gz -reads ${i}_2P.fq.gz 
--aligned "${i}_rRNA" --other "${i}_clean" 
--task 4 --threads 32 -v --paired_in --fastx

echo "Unmerging SortMeRNA filtered pairs for ${i}"
/home/nus/Desktop/NGS/sortmerna-2.1b/scripts/unmerge-paired-reads.sh "${i}_clean.fq" "${i}_clean1.fq"    "${i}_clean2.fq"

[Fabio Marroni]

Wow, I didn't even know about 4.2.0. I followed the link on the paper and got version 2.1. Just out of curiosity, are you using the binaries or you compiled from source code? I don't know if this could affect (it shouldn't but you never know).

[tianshenbio]

Hi, I used the binaries. It was released in Mar 2020. The reason I switched from 2.1 to the lastest one is simply that the 2.1 version always produces some corrupted files failing in fastqc (usually the read2 of a paired-end pair). They mentioned in the GitHub that this is solved in the new ver.

[Fabio Marroni]

I would either try another installation (source code or conda), or ask authors for help. The behavior is strange indeed.

[bojkem91]

I've found here that you should increase parameter -m " https://www.gitmemory.com/issue/biocore/sortmerna/231/619316128 "

[tianshenbio]

That's written by me :)

[Nifaste]

@tianshenbio to which value did you set -m option ? I have the same problem and setting -m to 12288 doesn't improve running time when compared to v2.1b.

[tianshenbio]

Hi， It was long time ago so I can’t really remember. But in the end I decided not to use the new version and went for the old version instead..

[Nifaste]

Thank you, it is what i'm going to do so !