AddOrReplaceReadGroups by Picard related error "Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded"

Question

[ravi.uhdnis]

Hi, I am encountering this error while adding 'Group information using Picard'. It would be great if anybody suggest me a solution for this error.

Command:

/usr/bin/java \
  -jar /usr/local/picard-tools-1.129/picard.jar \
  AddOrReplaceReadGroups \
  I=ETH002331.bwa.sam \
  O=ETH002331.bam \
  SO=coordinate \
  ID=2331 \
  PL=Illumina \
  PU=2331 \
  SM=2331 \
  LB=2331 \
  MAX_RECORDS_IN_RAM=10000000 \
  TMP_DIR=/san/illumina_two/tmp_rsindhu/

Output/Error:

[Tue Oct 06 10:02:55 EDT 2015] picard.sam.AddOrReplaceReadGroups INPUT=ETH002331.bwa.sam OUTPUT=ETH002331.bam SORT_ORDER=coordinate RGID=2331 RGLB=2331 RGPL=Illumina RGPU=2331 RGSM=2331 TMP_DIR=[/san/illumina_two/tmp_rsindhu] MAX_RECORDS_IN_RAM=10000000    VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 CREATE_INDEX=false CREATE_MD5_FILE=false
[Tue Oct 06 10:02:55 EDT 2015] Executing as rsindhu@node13 on Linux 2.6.18-164.15.1.el5 amd64; Java HotSpot(TM) 64-Bit Server VM 1.7.0_75-b13; Picard version: 1.129(b508b2885562a4e932d3a3a60b8ea283b7ec78e2_1424706677) IntelDeflater
INFO    2015-10-06 10:02:55    AddOrReplaceReadGroups    Created read group ID=2331 PL=Illumina LB=2331 SM=2331

INFO    2015-10-06 10:03:04    AddOrReplaceReadGroups    Processed     1,000,000 records.  Elapsed time: 00:00:08s.  Time for last 1,000,000:    8s.  Last read position: 3:93,470,726
INFO    2015-10-06 10:03:18    AddOrReplaceReadGroups    Processed     2,000,000 records.  Elapsed time: 00:00:22s.  Time for last 1,000,000:   13s.  Last read position: 11:54,692,144
INFO    2015-10-06 10:03:46    AddOrReplaceReadGroups    Processed     3,000,000 records.  Elapsed time: 00:00:50s.  Time for last 1,000,000:   27s.  Last read position: 1:9,626,979
INFO    2015-10-06 10:03:53    AddOrReplaceReadGroups    Processed     4,000,000 records.  Elapsed time: 00:00:57s.  Time for last 1,000,000:    7s.  Last read position: 15:36,735,360
INFO    2015-10-06 10:04:16    AddOrReplaceReadGroups    Processed     5,000,000 records.  Elapsed time: 00:01:20s.  Time for last 1,000,000:   22s.  Last read position: 12:68,352,108
INFO    2015-10-06 10:04:36    AddOrReplaceReadGroups    Processed     6,000,000 records.  Elapsed time: 00:01:40s.  Time for last 1,000,000:   19s.  Last read position: 1:204,840,100
INFO    2015-10-06 10:04:49    AddOrReplaceReadGroups    Processed     7,000,000 records.  Elapsed time: 00:01:53s.  Time for last 1,000,000:   13s.  Last read position: 1:163,618,828
INFO    2015-10-06 10:05:17    AddOrReplaceReadGroups    Processed     8,000,000 records.  Elapsed time: 00:02:21s.  Time for last 1,000,000:   28s.  Last read position: 6:129,383,605
INFO    2015-10-06 10:05:46    AddOrReplaceReadGroups    Processed     9,000,000 records.  Elapsed time: 00:02:50s.  Time for last 1,000,000:   28s.  Last read position: 4:87,904,801
INFO    2015-10-06 10:09:01    AddOrReplaceReadGroups    Processed    10,000,000 records.  Elapsed time: 00:06:05s.  Time for last 1,000,000:  194s.  Last read position: 7:102,840,920
INFO    2015-10-06 10:11:35    AddOrReplaceReadGroups    Processed    11,000,000 records.  Elapsed time: 00:08:39s.  Time for last 1,000,000:  154s.  Last read position: 13:24,434,813
INFO    2015-10-06 10:11:49    AddOrReplaceReadGroups    Processed    12,000,000 records.  Elapsed time: 00:08:53s.  Time for last 1,000,000:   14s.  Last read position: 7:92,219,814
INFO    2015-10-06 10:12:03    AddOrReplaceReadGroups    Processed    13,000,000 records.  Elapsed time: 00:09:07s.  Time for last 1,000,000:   13s.  Last read position: 3:70,583,119
INFO    2015-10-06 10:12:19    AddOrReplaceReadGroups    Processed    14,000,000 records.  Elapsed time: 00:09:23s.  Time for last 1,000,000:   16s.  Last read position: 3:14,135,029
INFO    2015-10-06 10:12:28    AddOrReplaceReadGroups    Processed    15,000,000 records.  Elapsed time: 00:09:32s.  Time for last 1,000,000:    9s.  Last read position: 5:178,739,899
INFO    2015-10-06 10:12:41    AddOrReplaceReadGroups    Processed    16,000,000 records.  Elapsed time: 00:09:45s.  Time for last 1,000,000:   12s.  Last read position: 8:112,973,946
INFO    2015-10-06 10:12:48    AddOrReplaceReadGroups    Processed    17,000,000 records.  Elapsed time: 00:09:52s.  Time for last 1,000,000:    7s.  Last read position: 1:125,179,853
INFO    2015-10-06 10:13:13    AddOrReplaceReadGroups    Processed    18,000,000 records.  Elapsed time: 00:10:17s.  Time for last 1,000,000:   25s.  Last read position: 2:71,028,202
INFO    2015-10-06 10:13:56    AddOrReplaceReadGroups    Processed    19,000,000 records.  Elapsed time: 00:11:00s.  Time for last 1,000,000:   42s.  Last read position: 9:60,641,119
[Tue Oct 06 10:29:47 EDT 2015] picard.sam.AddOrReplaceReadGroups done. Elapsed time: 26.87 minutes.
Runtime.totalMemory()=6657409024
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
    at htsjdk.samtools.util.StringUtil.stringToBytes(StringUtil.java:315)
    at htsjdk.samtools.SAMRecord.setReadString(SAMRecord.java:216)
    at htsjdk.samtools.SAMLineParser.parseLine(SAMLineParser.java:315)
    at htsjdk.samtools.SAMTextReader$RecordIterator.parseLine(SAMTextReader.java:248)
    at htsjdk.samtools.SAMTextReader$RecordIterator.next(SAMTextReader.java:236)
    at htsjdk.samtools.SAMTextReader$RecordIterator.next(SAMTextReader.java:212)
    at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:527)
    at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:501)
    at picard.sam.AddOrReplaceReadGroups.doWork(AddOrReplaceReadGroups.java:116)
    at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:187)
    at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
    at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)

[Adrian Pelin]

seems like you are running out of memory here.

[ravi.uhdnis]

Yes, that's written in the output message. Is addition of extra memory is the only alternative ?.

Answer 1

[cpad0112]

I am not sure about default memory picard uses. See if you can allocate more memory to JVM by using -Xmx option. Example command looks more like this: java -jar picard.jar -Xmx 4G <rest of arguments>. Please do search online on how to use -Xmx.

[ravi.uhdnis]

Thank you very much. Yes, I was missing this option in my command. Since I am working in WGS of Human Samples so this option is kind of necessary to use in order to handle large data sets while processing/aligning reads. Moreover, now I am running command like /usr/bin/java -Xmx40g -jar /usr/local/picard-tools-1.129/picard.jar ...., and it's working. Regards, Ravi.