VelvetOptimizer error - threads setup and more?

Question

[sheridan.christopher]

Hi everyone

I am currently trying to setup my submission script for VelvetOptimizer on our cluster and was a bit confused as to how the various parameters had to be setup. The plan was to spread the instances over 24 threads, so I initially had the following parameters:

• In the SLURM submission script: OMP_NUM_THREADS=24, OPENBLAS_NUM_THREADS=24, --cpus=24, --cpus-per-task=24, mem=256Gb
• In the VelvetOptimizer command line: -t 24

However it appears that this generated thread allocation issues (it maybe tried to allocate 24*24 threads...?), and the program eventually crashed with the following error messages (not sure if the thread allocation actually caused the crash though):

Use of uninitialized value in numeric ne (!=) at /home/umons/orgmb/csheridan/software/BioPerl/BioPerl-1.6.1/VelvetOptimiser.pl line 289.
Use of uninitialized value $maxScore in numeric gt (>) at /home/umons/orgmb/csheridan/software/BioPerl/BioPerl-1.6.1/VelvetOptimiser.pl line 290.
Use of uninitialized value in numeric gt (>) at /home/umons/orgmb/csheridan/software/BioPerl/BioPerl-1.6.1/VelvetOptimiser.pl line 290.
Aug 11 16:12:51 Hash value of best assembly by assembly score: 79
Aug 11 16:12:51 Optimisation routine chosen for best assembly: shortPaired
Aug 11 16:12:51 Looking for the expected coverage
Unable to open /gpfsuser/home/users/c/s/csheridan/data/auto_data_79/stats.txt for exp_cov determination.
 at /home/umons/orgmb/csheridan/software/BioPerl/BioPerl-1.6.1/VelvetOptimiser.pl line 838

Anyway, I then tried to correct this by setting the threads parameters as follows, but things appeared much slower with Velvet calculating only 3 hash values at a time:

• In the SLURM submission script: OMP_NUM_THREADS=8, OPENBLAS_NUM_THREADS=8, --cpus=24, --cpus-per-task=24, mem=256Gb
• In the VelvetOptimizer command line: -t 3

So my question is, how do you exactly set the threads parameters in order to gain from the use of OPENMP, and how can Velvet/Oases be optimally parallelised?

Thank you!

Answer 1

[Adrian Pelin]

Correct, it will use OMP_NUM_THREADS=8 per hash value and will calculate "-t 3" amount of hash values at the same time, so you should be using all 24 cores. The more hash values you try to calculate simultaneously, the more RAM you need, so play with the parameters for a bit, and make sure you do not go over your RAM. 256 GB RAM used to be a lot of RAM...

[sheridan.christopher]

Thanks Adrian.

I have used those parameters but still get a crash report:

****************************************************
           VelvetOptimiser.pl Version 2.2.5
****************************************************
Number of CPUs available: 64
Current free RAM: 246.374GB
Velvet OMP compiler setting: 8
Aug 11 16:28:37 Starting to check input parameters.
        Velveth parameter string OK.
Aug 11 16:28:37 Finished checking input parameters.
Velvet details:
        Velvet version: 1.2.10
        Compiled categories: 2
        Compiled max kmer length: 149
        Maximum number of velvetinstances to run: 3
Will run velvet optimiser with the following paramters:
        Velveth parameter string:
                -fastq -shortPaired -interleaved /home/umons/orgmb/csheridan/data/velvet-optimizer/merged_paired_shuffled_good140810.fastq -strand_specific -short -interleaved /home/umons/orgmb/csheridan/$
        Velveth start hash values:    29
        Velveth end hash value:         149
        Velveth hash step value:        10
        Velvetg minimum coverage cutoff to use: 0

        Read tracking for final assembly off.
Aug 11 16:28:37 Beginning velveth runs.
Logfile name: 11-08-2014-16-28-37_Logfile.txt
Aug 11 16:28:37         Running velveth with hash value: 29.
Aug 11 16:28:39         Running velveth with hash value: 39.
Aug 11 16:28:42         Running velveth with hash value: 49.

libgomp: Thread creation failed: Resource temporarily unavailable
Velveth failed!  Response:
1
Aug 11 16:33:00         Velveth with hash value 39 finished.
Waiting for thread 1 to complete.

libgomp: Thread creation failed: Resource temporarily unavailable
Aug 11 16:37:23         Velveth with hash value 29 finished.
Waiting for thread 2 to complete.
Waiting for thread 3 to complete.

libgomp: Thread creation failed: Resource temporarily unavailable
Aug 11 16:47:45         Velveth with hash value 49 finished.
Velveth failed to run! Must be a problem with file types, check by running velveth manually or by using -v option and reading the log file.

I have therefore attempted to run velveth on its own, and there is no issue. Do you have an idea of what may be causing this?

Thanks

[Adrian Pelin]

Yes, I think it is the lack of RAM. Did you monitor your RAM usage? Use htop or top. The lower the kmer, the more ram it needs.

[SES]

This does look like an issue with thread number, but it is kind of hard to debug. I recommend putting the following in your submission script:

export OMP_THREAD_LIMIT=1
export OMP_NUM_THREADS=1

and use -t 1 when you run VelvetOptimiser. If you have issues with those parameters it may indicate something else is going on. Also, that will tell you approximately how much RAM you will need per thread (it will vary by hash length though, so be aware of that).