bcl-convert multithreading optimisation

Question

[Juke34]

Hi,

I would like to better understand how threads are handled by bcl-convert (v3.8.2-12) but I don't find any clear information how to properly tune the parameters.

There are 4 options:

--bcl-num-decompression-threads
--bcl-conversion-threads
--bcl-num-compression-threads
--bcl-num-parallel-tiles

If I have 16 threads, should I set all this option to 16? If each step is independant I guess setting them to 16 would be fine, but I guess a least a part of the steps are overalping e.g ther is decompression, conversion, and compression on the fly.... So I should put 4 threads to each of the 4 steps?

The bcl-convert -h says:

  --bcl-num-parallel-tiles arg           # of tiles to process in parallel (default 1)
  --bcl-num-conversion-threads arg       # of threads for conversion (per tile, default # cpu threads)
  --bcl-num-compression-threads arg      # of threads for fastq.gz output compression (per tile, default # cpu threads,
                                         or HW+12)
  --bcl-num-decompression-threads arg    # of threads for bcl/cbcl input decompression (per tile, default half # cpu 
                                         threads, or HW+8. Only applies when preloading files)

while the online doc says that by default the threads are Determined dynamically. I have tried and it is true it is dinamically determined, but this is a huge problem in my nextflow pipeline because it uses all the CPUs while I need some for other purposes. So I definilty want to set the threads to use myself.

P.S: Acutally I had the same thread tuning problem with bcl2fastq, does someone has a good tunnning with the 3 parameters from bcl2fastq that I can get inspired from? I can apply the same to bcl-convert

Cheers

Answer 1

[GenoMax]

We set the individual threads for those options so they add up to actual number of cores you have available for use. You don't want to have contention amongst the threads. Some of this depends on speed/throughput of your storage system. It is reading/writing that takes more time (so consider assigning more threads to compression/decompression) than the actual basecalling. You can probably tweak things but I am not sure if it is worth spending a lot of time on this.

[Juke34]

decompression (bcl-num-decompression-thread) seems to be less demanding default half # cpu threads

[GenoMax]

Do a bit of local testing since much of this will depend on speed/efficacy of your storage.