Abyss stuck for past 48 days in generating adjency

Question

[way2manmohan]

I am running denovo whole genome assembly using Illumina NextSeq paired-end data of about 300 gb for a eukaryotic genome of approx 1.45 gb(genome size). My server configuration is 512gb RAM and 7.5 tb hard disc with 64 processor. My problem is that the assembly got stuck in the step 'generating adjency' from the past 30-35 days. I have started the assembly on 7th of May and its around 48 days since its running. Now I am little bit confused whether to abort the assembly or should I continue. Kindly guide me. I am not using MPI option since we don't have cluster in our machine. I would like to know is there any process or code to enhance the speed of assembly.

[benv]

Hi @way2manmohan,

Wow... more than 30 days! Something is wrong for sure. (I have assembled human genomes with the MPI version, and the first stage typically takes about 24 hours.)

@cyril-cros may be right about running out of RAM (below). It is good advice to watch the process in top or htop. By "stuck", do you mean that you are no longer seeing any new messages in the log output? It would be helpful to post your full log output to a github gist and link to it here. Also, please enable the verbose option, if possible (add v=-v to the abyss-pe command line).

It is possible to use the MPI version even on a single machine, and it will speed things up greatly. When using the non-MPI version, you are only using 1 of 64 available cores. To use the MPI version you just need to have a recent OpenMPI library installed on your system (and also the OpenMPI development headers.) If you have a system administrator, they should be able to do that for you very easily.

Answer 1

[cyril-cros]

Check out this thread. 512Gb RAM might not be enough. You can try running some commands like htop or cat /proc/meminfo to see if this is the case. Also, do you have any log file? For another time, run abyss with the -v(erbose) flag.

Answer 2

[Shaun Jackman]

I'm pretty sure that you've exceed the available memory. If you run ABySS with abyss-pe v=-v it will tell you how much memory each node is using. If it's already running, run top or ps aux |grep ABYSS. For a large genome, be sure to use the MPI version by specifying abyss-pe np=12 where 12 is the number of cores to use. Ensure that Google Sparsehash is being used. In the directory where ABySS was compiled, run the following command:

❯❯❯ grep SPARSE config.h
#define HAVE_GOOGLE_SPARSE_HASH_MAP 1

Cheers,
Shaun