Memory usage of picard Samsort

Question

[boymin2020]

I want to use SortSam of picard to coordinate sort and index the individual WGS data (e.g. 73G). The memory size of one node in our cluster is near 126.2G. I even set the memory size in qsub command to 110G and distributed 100G to java, it still gave me the following error message : Exception in thread "main" htsjdk.samtools.util.RuntimeIOException: java.io.IOException: No space left on device

What should I do ? Any advice will be appreciated.

[boymin2020]

I have fixed the problem by adding the optional parameter TMP_DIR. Thanks for help.

Answer 1

[Pierre Lindenbaum]

• use the classical samtools
• or enhance the memory of java using -Xmx
• or/and change the parameters of picard like MAX_RECORDS_IN_RAM

When writing SAM files that need to be sorted, this will specify the number of records stored in RAM before spilling to disk. Increasing this number reduces the number of file handles needed to sort a SAM file, and increases the amount of RAM needed. Default value: 500000. This option can be set to 'null' to clear the default value.

[Pierre Lindenbaum]

, it still gave me the following error message : Exception in thread "main" htsjdk.samtools.util.RuntimeIOException: java.io.IOException:

what @dariober said.

Use the standard option

TMP_DIR=

to change the default dir

Answer 2

[dariober]

java.io.IOException: No space left on device

I don't think the problem is with the memory but rather with the disk space. Picard writes temporary files to /tmp which usually is not a very large partition and it gets filled up easily. Try setting the TMP_DIR option to somewhere else, e.g. the current working directory (which is where samtools sort writes its tmp files).

Answer 3

[swbarnes2]

Simple not-very-clever solution...use grep or something to split out the files by chromosome, sort those individually, then cat them back together (with the orignal headers)

Answer 4

[tamas.malkocs]

For me, setting TMP_DIR (--TMP_DIR /home/tmp) didn't work in picard 2.21.1. What solved the issue was setting the environmental variable : export _JAVA_OPTIONS=-Djava.io.tmpdir=/home/tmp .

[ATpoint]

Guys, just use samtools sort which won't get you into trouble with any of that.