Internal memory allocator is full!

Question

[cmckay]

Hello, thank you for making the Minia software. I hope you can help me get it working with my dataset, which is 24GB (compressed) of illumina reads. The sample is an environmental metagenome, so is very complex. I selected an estimated_genome_size of 243000000000 not for biological reasons, but to try to compel minia to use a goodly portion of the 240GB ram on this machine. As you can see at the end, Minia crashed. If you have any input or ideas, I'd appreciate the help.

# minia all_paired.below.khmer.fasta.gz 31 3 243000000000 all_paired
estimated values: nbits Bloom 1466351878144, nb FP 3526388992, max memory 174802 MB
taille cell 16 
Available disk space in /mnt: 548951 MB
Sequentially counting ~409503 MB of kmers with 2 partition(s) and 5 passes using 1 thread(s), ~174802 MB of memory and ~95175 MB of disk space
| First step: Converting input file into Binary format                                               |
[----------------------------------------------------------------------------------------------------]
| Counting kmers                                                                                     |
100    %     elapsed:    262 min 24    sec      estimated remaining:      0 min 0     sec 

Saved 4571564210 solid kmers
-------------------Counted kmers time Wallclock  16624.1 s

------------------ Counted kmers and kept those with abundance >=3,     
nelem 4571564210 nbits 6.03437 
freesize 27586508800 estimated_BL1_freesize  3288.6 MB of memory for the main Bloom structure (6.03437 bits/kmer)
binary pass 
Insert solid Kmers in Bloom 4571560000
Inserted 4571564210 solid kmers in the bloom structure.
 Writing positive Bloom Kmers 4571560000
10588954139 kmers written
-------------------Write all positive kmers time Wallclock  8596.82 s
max size for this hash is 2^32, resuming with max value 
0 partitions will be needed
Build Hash table 4290770000Internal memory allocator is full!
Segmentation fault (core dumped)

[edrezen]

Hello,

I think you are using an old version of minia. You should have a try to the new one here (much faster with improved result quality); there are chances that you won't face the same memory issues with this new version.

[Dan D]

Edited to add "minia" tag.