Problem using PicardTools MarkDuplicates: SortingCollection: There is not enough memory per file for buffering. Reading will be unbuffered.

Question

[pengxw1990]

MarkDuplicates -I KPGP-00001.final.bam -O KPGP-00001.markdup.bam -M KPGP-00001.markdup.metrics -CREATE_INDEX true -REMOVE_DUPLICATES true -ASSUME_SORTED true -VALIDATION_STRINGENCY LENIENT -TMP_DIR /mnt/12T/tmp

OpenJDK 64-Bit Server VM 1.8.0_275-b01; Deflater: Intel; Inflater: Intel; Provider GCS is not available; Picard version: 2.24.2

INFO    2021-02-23 01:53:53 MarkDuplicates  After buildSortedReadEndLists freeMemory: 49282740792; totalMemory: 49578770432; maxMemory: 49578770432
INFO    2021-02-23 01:53:53 MarkDuplicates  Will retain up to 1549336576 duplicate indices before spilling to disk.
INFO    2021-02-23 01:54:34 MarkDuplicates  Traversing read pair information and detecting duplicates.

WARNING 2021-02-23 01:57:49 SortingCollection   There is not enough memory per file for buffering. Reading will be unbuffered.**

INFO    2021-02-23 02:31:19 MarkDuplicates  Traversing fragment information and detecting duplicates.
INFO    2021-02-23 03:19:01 MarkDuplicates  Sorting list of duplicate records.
INFO    2021-02-23 03:20:05 MarkDuplicates  After generateDuplicateIndexes freeMemory: 36950789560; totalMemory: 49681006592; maxMemory: 49681006592

[Medhat]

You can increase memory like java -Xms10G -Xmx20G -jar /path/to/picard-tools/picard.jar MarkDuplicates.

min: 10

max: 20

[pengxw1990]

Thanks, I will try it.

[GenoMax]

This is a WARNING. You should still be able to get the output.

[pengxw1990]

Yeah, i did get a output file, but it's not complete. Thanks a lot.

[GenoMax]

How do you know that? Were there additional errors?

[pengxw1990]

hah, you are right! for hours later, this command started running again! might it just had a rest....