Minia: Pool allocation failed for 68766984 bytes (kmers alloc), mainbuffer is null?.

Question

[Beuss]

Hello,

I'm trying to assemble a plant genome (1.1 Gbp) from 38 illumina reads datasets (pair end and mate pair) with a coverage of 150x.

The version of Minia that I use is :

Minia 3, git commit 67ac2ad.

The parameters used are :

minia -in inMinia.txt -out plant_79 -kmer-size 79 -max-memory 500000 -nb-cores 11

The program run on a node with 500 Gb of RAM.

Minia is not able to run the full analysis. It stoped and throw the following errors:

...
      [DSK: Pass 1/1, Step 1: partitioning     ]  43   %   elapsed: 271 min 51 sec   remaining: 360 min 21 sec   cpu: 404.0 %   mem: [7044, 29535, 30387] MB 
      [DSK: Pass 1/1, Step 1: partitioning     ]  44   %   elapsed: 280 min 25 sec   remaining: 356 min 53 sec   cpu: 402.5 %   mem: [3643, 29535, 30387] MB 
      [DSK: Pass 1/1, Step 1: partitioning     ]  45   %   elapsed: 288 min 46 sec   remaining: 352 min 56 sec   cpu: 400.9 %   mem: [1606, 29535, 30387] MB 
      [DSK: Pass 1/1, Step 1: partitioning     ]  46   %   elapsed: 297 min 9  sec   remaining: 348 min 49 sec   cpu: 399.4 %   mem: [ 748, 29535, 30387] MB 
      [DSK: Pass 1/1, Step 2: counting kmers   ]  46.4 %   elapsed: 300 min 6  sec   remaining: 347 min 5  sec   cpu: 399.1 %   mem: [ 127, 29535, 30387] MB EXCEPTION: Pool allocation failed for 76422408 bytes (kmers alloc), mainbuffer is null?. Current usage is 76422424 and capacity is 524288001200
      Pool allocation failed for 68766984 bytes (kmers alloc), mainbuffer is null?. Current usage is 145189416 and capacity is 524288001200
      Pool allocation failed for 71142984 bytes (kmers alloc), mainbuffer is null?. Current usage is 216332408 and capacity is 524288001200
      Pool allocation failed for 73314816 bytes (kmers alloc), mainbuffer is null?. Current usage is 289647232 and capacity is 524288001200
      Pool allocation failed for 61646136 bytes (kmers alloc), mainbuffer is null?. Current usage is 351293384 and capacity is 524288001200
      Pool allocation failed for 81025200 bytes (kmers alloc), mainbuffer is null?. Current usage is 432318592 and capacity is 524288001200
      Pool allocation failed for 80461512 bytes (kmers alloc), mainbuffer is null?. Current usage is 512780120 and capacity is 524288001200
      Pool allocation failed for 66645312 bytes (kmers alloc), mainbuffer is null?. Current usage is 579425440 and capacity is 524288001200
      Pool allocation failed for 74095848 bytes (kmers alloc), mainbuffer is null?. Current usage is 653521304 and capacity is 524288001200
      Pool allocation failed for 75911232 bytes (kmers alloc), mainbuffer is null?. Current usage is 729432544 and capacity is 524288001200
      Pool allocation failed for 64741440 bytes (kmers alloc), mainbuffer is null?. Current usage is 794174000 and capacity is 524288001200

The problem seems to be related with the memory, but, because of the hardware configuration, I'm don't think that the node is lacking of RAM.

Do you any ideas for fixing this ?

Thanks in advance for answers.

Answer 1

[guillaume.rizk]

Hi,

Can you try again with -max-memory 200000 ? I think it tried to allocate the 500 GB you specified and failed since it is exactly the system ram.

[Beuss]

Hi,

I'm think your are right. Before your answer, I launched new assemblies with the -max-memory option set to 0 (no limit I presumed). For now, 3 of the 5 assemblies that I launched finished with no problems. I'm waiting for the results of the two last before confirming the solution.

Thank you for the help.

[guillaume.rizk]

I just checked, when you set -max-memory option to 0, it will default to 5 GB max ( or system memory / 2 if system has less than 5 GB). It should run fine, but in your case, if you need to run it again, it should be a little faster with more memory, maybe -max-memory 200000

[Beuss]

Thank for the advise ! It will be applied for the next runs.

[Beuss]

Ok the problem is sovled ! Thanks again.