malt-build and Java heap Space

Question

[stu111392]

Hey everybody,

At the moment I try to index a bunch of sequences with malt. I've indexed all viral proteins from refseq without a problem. But now I wanted to index all bacterial ref-seq sequences at once. The file is about 9.8gb great and contains 5242 sequences. I used the command:

./malt-build -L /ifs/data/nfs_share/sukmb241/home.julian/malt/MEGAN5-academic-license.txt --input references/bacteriall_nucleotide --sequenceType DNA --index index/bacteriall_nr/

But after allocating the hash table the program spits out the following error:

Allocating: 75.6 GB
java.lang.OutOfMemoryError: Java heap space
        at java.nio.HeapByteBuffer.<init>(Unknown Source)
        at java.nio.ByteBuffer.allocate(Unknown Source)
        at jloda.E.H.<init>(Unknown Source)
        at jloda.E.D.<init>(Unknown Source)
        at malt.data.O.A(Unknown Source)
        at malt.MaltBuild.run(Unknown Source)
        at malt.MaltBuild.main(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.exe4j.runtime.LauncherEngine.launch(Unknown Source)
        at com.install4j.runtime.launcher.UnixLauncher.main(Unknown Source)

I changed the -Xmx parameter in the vmoptions files as described in the manual and my last try to get this to work was with 120g but it crashed with the exact same error message as above. At the moment I have no way to increase the memory level. Has anybody experience with the indexing of greater files in Malt? Did you encounter the same problem?

With best regards and thanks in advance,

Julian

Answer 1

[Ram]

The manual here says that maximum memory availability is set at setup time.

Quote from manual:

The most important performance-related option is the maximum amount of memory that malt-build is allowed to use. This cannot be set from within the program but rather is set during installation of the software

There also seems to be another option:

The installation dialog will ask how much memory the program may use. Please set this variable carefully. If the amount needs to be changed after installation, then this can be done by editing the files ending on vmoptions in the installation directory.

Please, always read the manual before asking on forums.

[stu111392]

I read the manual carefully and adjusted the memory parameters in vmop files up to 100gb RAM. This was already done and I therefor I asked the question here. So has anyone out there a helpful answer?

[Ram]

1. You did not mention that you already tried this. If you did, I would not have suggested this.
2. There's no need to be dismissive. Even if my answer were not relevant (which is not the case here because of point #1), you should acknowledge it better.
3. Your response belongs in a comment to my answer, not an answer of its own. I've moved it to the appropriate location now.

Answer 2

[Brian Bushnell]

If you have enough memory, you might try something this:

export JAVA_OPTIONS="-Xmx200g"

Not sure about malt but in general it will override any subsequent attempt to set the -Xmx flag.

[stu111392]

Thank you for that. I'll try it out tomorrow.

Answer 3

[stu111392]

And maybe let me rephrase the question a bit. I'm not entirely sure if this is just a pure problem of RAM or a Java error and therefore I posted the question here. Because even if I give the task 100gb of RAM it says after the hash table that 75,6GB are allocated. And therefor I thought it has another reason.

[Brian Bushnell]

Java does not use 100% of the memory for data; a large fraction is reserved for garbage-collection. Being able to use 75% sounds pretty typical.

[stu111392]

So your suggestion would be to increase the overall amount of memory?

[Brian Bushnell]

Unless you have a reason not to, just give Java all the physical memory of the machine. "java.lang.OutOfMemoryError: Java heap space" means that it was not given enough memory.

[Ram]

Edit your question and add all relevant information in there please. People will then not have to read through all the answers and comments to give you appropriate input.

[stu111392]

I changed it. Hope that the problem is clearer now.