BBMap genome indexing - java issue?

Question

[CAnna]

Hi,

I want to remove human contaminants from gut metagenomes. For this I am using BBMap. The first step is to index the contaminant genome (human genome). I am following the exact procedure as described here, which I cross validated by finding the same commands described on other pages (biostars, github).

My command:

bbmap.sh -Xmx24G ref=hg19_main_mask_ribo_animal_allplant_allfungus.fa.gz

is simply not working. My script outputs an output and error files. The output file is empty, the error files simply writes the command

java -Djava.library.path=/exports/csce/eddie/biology/groups/mcnally/camille/programs/bbmap/jni/ -ea -Xmx24G -cp /exports/csce/eddie/biology/groups/mcnally/camille/programs/bbmap/current/ align2.BBMap build=1 overwrite=true fastareadlen=500 -Xmx24G ref=hg19_main_mask_ribo_animal_allplant_allfungus.fa.gz

and I have no other output.

I can't figure out the problem. Am I using the command wrong? Or is it an issue with my java? I am using my university cluster.

Thanks, Camille

[GenoMax]

If you are using a cluster are you asking for a corresponding amount of RAM from the job scheduler? Are you using Java 7/8? You should also look at the log files of your job scheduler and post the errors present here.

[CAnna]

Thanks for your reply

Yes I am specifying -l h_vmem=24G on the job scheduler

I am using Java 8 (java/jdk/1.8.0)

And the log file of the job scheduler is here (if this is what you meant?)

qname        eddie
hostname     node2c03.ecdf.ed.ac.uk
group        eddie_users
owner        s1687811
project      uoe_baseline
department   defaultdepartment
jobname      index_bbmap
jobnumber    1976327
taskid       undefined
account      sge
priority     0
qsub_time    Sun Apr  8 22:16:04 2018
start_time   Sun Apr  8 22:16:13 2018
end_time     Sun Apr  8 22:16:13 2018
granted_pe   NONE
slots        1
failed       0
exit_status  1
ru_wallclock 0
ru_utime     0.010
ru_stime     0.007
ru_maxrss    6128
ru_ixrss     0
ru_ismrss    0
ru_idrss     0
ru_isrss     0
ru_minflt    4243
ru_majflt    0
ru_nswap     0
ru_inblock   0
ru_oublock   72
ru_msgsnd    0
ru_msgrcv    0
ru_nsignals  0
ru_nvcsw     50
ru_nivcsw    67
cpu          0.017
mem          0.000
io           0.000
iow          0.000
maxvmem      0.000
arid         undefined

[GenoMax]

That is not telling us much.

I am running as indexing operation with the same file to test and so far no problem. I am using 30g of RAM. Will update later as to what happens.

[CAnna]

Ha, this is tricky, the thing is that I do not have any output/error to suggest what is wrong. I don't think it is a memory issue since maxvmem if I recall says how much memory has been used at max and this says 0. Looks like it did not run at all?

[GenoMax]

You could start running it outside the job scheduler to make sure bbmap is working fine (be ready to kill the job). I was able to complete the indexing without any problems. I did use 30g of RAM.

Answer 1

[CAnna]

Arg, I figured it out. For the sake of potential beginners having this kind of issue, this was related to my shell script.

If you do specify a value for -Xmx it needs to be less than the amount of virtual memory that you reserve for your job. In my case I had reserved 30GB of vmem, so should set -Xmx to something like:

-Xmx15G

as an example.

However, all this is moot if you are running Java 1.8. You shouldn't specify a value for -Xmx for this newer version of Java. You should only specify a value for this parameter if you are running Java 1.7 or lower.

So, in summary, re-running your job with this line with Java 1.8

bbmap.sh ref=hg19_main_mask_ribo_animal_allplant_allfungus.fa.gz

or with Java 1.7

bbmap.sh -Xmx15G ref=hg19_main_mask_ribo_animal_allplant_allfungus.fa.gz

You might have to experiment with the value of -Xmx depending on the genome you are eindexing but the key thing is it must be less than the amount of virtual memory you reserve for your job.

[GenoMax]

While I don't fully understand this explanation (never had this issue with Java 7/8) as long as it worked for you that is good. This may be a local configuration specific thing. Thank you for coming back and posting a resolution.