SPAdes error during assembly, error code -6

Question

[marongiu.luigi]

Dear all,

I ran SPAdes v3.13.0 on my Ubuntu machine with 16 threads and 64 Gb RAM (it used up to 25 so there was room to spare) using fastq files with the command spades.py --debug -1 ${inFile}_1.fq.gz -2 ${inFile}_2.fq.gz -o $OUT where {inFile} were the input fastq and $OUT was the output folder. I got this error:

Command line: /home/gigiux/src/SPAdes/bin/spades.py --debug -1  /home/gigiux/servers/chirexpsrv3_a32/LUIGI_HIPO32_nls/data/A1/A1-N_trimP_1.fq.gz    -2  /home/gigiux/servers/chirexpsrv3_a32/LUIGI_HIPO32_nls/data/A1/A1-N_trimP_2.fq.gz    -o  /home/gigiux/Downloads/h32/confirm/rslt/A1-N    

System information:
  SPAdes version: 3.13.0
  Python version: 2.7.15
  OS: Linux-4.15.0-55-generic-x86_64-with-Ubuntu-18.04-bionic

Output dir: /home/gigiux/Downloads/h32/confirm/rslt/A1-N
Mode: read error correction and assembling
Debug mode is turned ON

Dataset parameters:
  Multi-cell mode (you should set '--sc' flag if input data was obtained with MDA (single-cell) technology or --meta flag if processing metagenomic dataset)
  Reads:
    Library number: 1, library type: paired-end
      orientation: fr
      left reads: ['/home/gigiux/servers/chirexpsrv3_a32/LUIGI_HIPO32_nls/data/A1/A1-N_trimP_1.fq.gz']
      right reads: ['/home/gigiux/servers/chirexpsrv3_a32/LUIGI_HIPO32_nls/data/A1/A1-N_trimP_2.fq.gz']
      interlaced reads: not specified
      single reads: not specified
      merged reads: not specified
Read error correction parameters:
  Iterations: 1
  PHRED offset will be auto-detected
  Corrected reads will be compressed
Assembly parameters:
  k: automatic selection based on read length
  Repeat resolution is enabled
  Mismatch careful mode is turned OFF
  MismatchCorrector will be SKIPPED
  Coverage cutoff is turned OFF
Other parameters:
  Dir for temp files: /home/gigiux/Downloads/h32/confirm/rslt/A1-N/tmp
  Threads: 16
  Memory limit (in Gb): 62


======= SPAdes pipeline started. Log can be found here: /home/gigiux/Downloads/h32/confirm/rslt/A1-N/spades.log


===== Read error correction started. 


== Running read error correction tool: /home/gigiux/src/SPAdes/bin/spades-hammer /home/gigiux/Downloads/h32/confirm/rslt/A1-N/corrected/configs/config.info

  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  75)   Starting BayesHammer, built from refs/heads/spades_3.13.0, git revision 8ea46659e9b2aca35444a808db550ac333006f8b
  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  76)   Loading config from /home/gigiux/Downloads/h32/confirm/rslt/A1-N/corrected/configs/config.info
  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  78)   Maximum # of threads to use (adjusted due to OMP capabilities): 16
  0:00:00.000     4M / 4M    INFO    General                 (memory_limit.cpp          :  49)   Memory limit set to 62 Gb
  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  86)   Trying to determine PHRED offset
  0:00:00.022     4M / 4M    INFO    General                 (main.cpp                  :  92)   Determined value is 33
  0:00:00.022     4M / 4M    INFO    General                 (hammer_tools.cpp          :  36)   Hamming graph threshold tau=1, k=21, subkmer positions = [ 0 10 ]
  0:00:00.022     4M / 4M    INFO    General                 (main.cpp                  : 113)   Size of aux. kmer data 24 bytes
     === ITERATION 0 begins ===
  0:00:00.023     4M / 4M    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index
  0:00:00.023     4M / 4M    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 256 files using 16 threads. This might take a while.
  0:00:00.023     4M / 4M    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 1024
  0:00:00.023     4M / 4M    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 1.29159 Gb
  0:00:00.023     4M / 4M    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 262144
  0:00:00.049     8G / 8G    INFO   K-mer Splitting          (kmer_data.cpp             :  97)   Processing /home/gigiux/servers/chirexpsrv3_a32/LUIGI_HIPO32_nls/data/A1/A1-N_trimP_1.fq.gz
  0:00:21.171     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 107)   Processed 4464627 reads
  0:00:42.335     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 107)   Processed 8911972 reads
  0:01:03.754     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 107)   Processed 13386745 reads
  0:01:24.912     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 107)   Processed 17857950 reads
  0:01:46.146     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 107)   Processed 22319489 reads
  0:02:07.461     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 107)   Processed 26791350 reads
  0:02:30.416     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 107)   Processed 31275630 reads
  0:02:52.083     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 107)   Processed 35748235 reads
  0:03:13.444     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 107)   Processed 40210252 reads
  0:03:34.733     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 107)   Processed 44699957 reads
  0:03:56.218     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 107)   Processed 49195072 reads
  0:05:21.804     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 107)   Processed 67164448 reads
  0:10:31.166     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             :  97)   Processing /home/gigiux/servers/chirexpsrv3_a32/LUIGI_HIPO32_nls/data/A1/A1-N_trimP_2.fq.gz
  0:10:52.349     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 107)   Processed 136323345 reads
  0:21:09.287     8G / 9G    INFO   K-mer Splitting          (kmer_data.cpp             : 112)   Total 263918634 reads processed
  0:21:09.649    64M / 9G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.
  0:27:53.987    64M / 9G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 1510027312 kmers in total.
  0:27:53.987    64M / 9G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.
  0:28:21.826    64M / 9G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices
  0:30:13.149   776M / 9G    INFO    General                 (kmer_index_builder.hpp    : 150)   Merging final buckets.
  0:31:19.869   776M / 9G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 700226586 bytes occupied (3.70974 bits per kmer).
  0:31:19.870   776M / 9G    INFO   K-mer Counting           (kmer_data.cpp             : 356)   Arranging kmers in hash map order
  0:33:10.090    23G / 23G   INFO    General                 (main.cpp                  : 148)   Clustering Hamming graph.
  1:18:25.420    23G / 23G   INFO    General                 (main.cpp                  : 155)   Extracting clusters
<jemalloc>: Error in malloc(): out of memory. Requested: 24, active: 65968013312


== Error ==  system call for: "['/home/gigiux/src/SPAdes/bin/spades-hammer', '/home/gigiux/Downloads/h32/confirm/rslt/A1-N/corrected/configs/config.info']" finished abnormally, err code: -6

In case you have troubles running SPAdes, you can write to [e-mail]
or report an issue on our GitHub repository github.com/ablab/spades
Please provide us with params.txt and spades.log files from the output directory.

Essentially it is en error -6 but I used SPAdes on other files and it worked. Also, I used THESE files for a BWA mapping and they worked. What would be the error?

Also, I understand it is possible to run SPAdes with BAM files. What would be the syntax? Can I use deduplicated BAMs?

Thank you

[Buffo]

May be the option --debug is causing it falls out of memory

[marongiu.luigi]

Nope, I ran without it and I got the same problem. I added --debug to try and understand what was the problem. anyway, the memory consumption was still 25 Gb so I should have ~40 Gb spared. Also, it was not a single pair of files: the same error came from different patients.

[Buffo]

It is a simply memory limit, try increasing the number of threads as posible -t and set the memory limit -m (64), but it looks like you will need much more memory.

[marongiu.luigi]

I'll try but weren't these already set? from the log: Other parameters: Dir for temp files:.../A1-N/tmp Threads: 16 Memory limit (in Gb): 62

[marongiu.luigi]

Could be instead this the problem?

===== Read error correction started. 
== Running read error correction tool: /home/gigiux/src/SPAdes/bin/spades-hammer /home/gigiux/Downloads/h32/confirm/rslt/A1-N/corrected/configs/config.info

[Corentin]

Are you sure you have access to enough memory on your machine / server ?

Answer 1

[h.mon]

You are running out of memory. The error message says SPAdes is already using 64Gb, then needed more and requested an additional 24Gb. As these 24Gb are not available due to your server limitations, SPAdes crashes.

Essentially it is en error -6 but I used SPAdes on other files and it worked. Also, I used THESE files for a BWA mapping and they worked. What would be the error?

The fastq files are fine, mapping in general uses a let less memory than assembling. As for why you can assemble some datasets and not this one, certain genome characteristics may cause memory use to increase a lot during assembly, such as repetitive material content.

[marongiu.luigi]

you are right, after setting the threads and memory values, i got the error <jemalloc>: Error in malloc(): out of memory. Requested: 24, active: 63858278400. I need a bigger cluster...