rsem-calculate-expression ran out of memory

Question

[moxu]

I first built the terence with the following command

rsem-prepare-reference --bowtie reference_hg19.fa hg19_ref

And then ran the following:

rsem-calculate-expression --paired-end PairedEnd1.fastq PairedEnd2.fastq hg19_ref my_test_sample

After running for about 20 hours, all I got was the following output:

---------------------------------------------------- start ----------------------------------------------------
bowtie -q --phred33-quals -n 2 -e 99999999 -l 25 -I 1 -X 1000 -p 1 -a -m 200 -S hg19_ref -1 PairedEnd1.fastq -2 PairedEnd2.fastq | samtools view -S -b -o my_test_sample.temp/my_test_sample.bam -
Warning: Exhausted best-first chunk memory for read NS500144:471:HVFYMBGXX:1:12107:15053:9521 1:N:0:CAGAGAGG/1 (
patid 4851744); skipping read
Warning: Exhausted best-first chunk memory for read NS500144:471:HVFYMBGXX:1:21109:16613:17366 1:N:0:CAGAGAGG/1 
(patid 13368267); skipping read
Warning: Exhausted best-first chunk memory for read NS500144:471:HVFYMBGXX:2:13106:13072:15868 1:N:0:CAGAGAGG/1 
(patid 33771310); skipping read
Warning: Exhausted best-first chunk memory for read NS500144:471:HVFYMBGXX:2:21111:9396:9031 1:N:0:CAGAGAGG/1 (p
atid 38469910); skipping read
Warning: Exhausted best-first chunk memory for read NS500144:471:HVFYMBGXX:2:23211:19862:18298 1:N:0:CAGAGAGG/1 
(patid 48288092); skipping read
---------------------------------------------------- end ----------------------------------------------------

So I cancelled it.

It was ran on a server which had over 300GB memory, so I don't think it's the problem of the physical memory we have. Anyone has any idea?

Thanks a lot in advance!

-- m.x.

[Farbod]

Hi, maybe you can search it here : https://groups.google.com/forum/#!forum/rsem-users

[moxu]

Very helpful, and I joined the RSEM users google group. Thank you!

[moxu]

I reran rem-calculate-expression with added parameter "-p 8", but go the same warning for the "chunk memory".

[GenoMax]

-p option is for cores to use. Option you want to change would be --bowtie-chunkmbs <int> Default is 64 MB so set it to something bigger.

[moxu]

What <int> would you suggest? Some people use 256, but why not set it to even bigger, say, 1024?

Thanks!

Answer 1

[davedeto]

Not sure if you found your answer yet, but the issue is probably that you are using the whole genome, and not just the transcriptome (expressed transcripts only).

Read the examples here (http://deweylab.biostat.wisc.edu/rsem/rsem-prepare-reference.html) when you prepare your reference. Either the reference file already needs to be split into transcripts (if you open it, you'll see one fasta header, ">", for each transcript, not each chromosome), or else you need to call it with the -gtf option, and supply a GTF file that tells RSEM where the genes are.

[moxu]

--gtf & -p made the processing work. Thanks a lot!