How to convert a for loop to a Job-array in LSF cluster

Question

[LDT]

I have 100 files, and I want to parallelise my submission to save time instead of running jobs one by one. How can I change this script to a Job-array in LSF using bsub submission system?

#BSUB -J ExampleJob1         #Set the job name to "ExampleJob1"
#BSUB -L /bin/bash           #Uses the bash login shell to initialize the job's execution environment.
#BSUB -W 2:00                #Set the wall clock limit to 2hr
#BSUB -n 1                   #Request 1 core
#BSUB -R "span[ptile=1]"     #Request 1 core per node.
#BSUB -R "rusage[mem=5000]"  #Request 5000MB per process (CPU) for the job
#BSUB -M 5000                #Set the per process enforceable memory limit to 5000MB.
#BSUB -o Example1Out.%J      #Send stdout and stderr to "Example1Out.[jobID]"

path=./home/

for each in *.bam 
do 
samtools coverage ${each} -o ${each}_coverage.txt
done

Thank you for your time; any help is appreciated. I am a starter at LSF and quite confused

[5heikki]

How about just submitting jobs in a for loop, i.e. remove the loop from the script and replace ${each} with $1

[LDT]

I ll try this one 5heikki, it seems very promising. I am sure thought how i can define the range to run only 10 jobs at a time

[5heikki]

It shouldn't be your responsibility. Whoever setup the queue manager should be responsible for such stuff. As an example, you submit 10k jobs, but only 10 run in parallel (if there are available slots) because that's how the queue manager was setup..

Answer 1

[Pierre Lindenbaum]

ok, second answer with a Makefile using option -j

SHELL=/bin/bash
BAMS=$(shell ls -1 /path/to/*.bam)
define run

$$(addsuffix .out,$(1)): $(1)
    samtools coverage  -o $$@ $$<
endef

all: $(addsuffix .out,$(basename ${BAMS}))

$(eval $(foreach B,${BAMS},$(call run,${B})))

and run with:

#BSUB -J ExampleJob1         #Set the job name to "ExampleJob1"
#BSUB -L /bin/bash           #Uses the bash login shell to initialize the job's execution environment.
#BSUB -W 2:00                #Set the wall clock limit to 2hr
#BSUB -n 10                   #Request 10 core <====================
#BSUB -R "span[ptile=1]"     #Request 1 core per node.
#BSUB -R "rusage[mem=5000]"  #Request 5000MB per process (CPU) for the job
#BSUB -M 5000                #Set the per process enforceable memory limit to 5000MB.
#BSUB -o Example1Out.%J      #Send stdout and stderr to "Example1Out.[jobID]"


cd /path/to/your/makefile/dir && make -j 10

Answer 2

[Pierre Lindenbaum]

use a workflow manager like nextflow. e.g (not tested)

params.bams="NO_FILE"

workflow  {
        ch1= ST_COVERAGE(params,Channel.fromPath(params.bams).splitText().map{it.trim()})
        MAKE_LIST(params,ch1.output.collect())
        }
process ST_COVERAGE {
tag "${bam}"
input:
     val(meta)
     val(bam)
output:
    path("coverage.txt"),emit:output
script:
"""
samtools coverage -o coverage.txt "${bam}"
"""
}

process MAKE_LIST {
input:
   val(meta)
   val(L)
output:
  path("all.list"),emit:output
script:
"""
cat << EOF > all.list
${L.join("\n")}
EOF
"""
}

and run, something like

nextflow run -C lsf.config -resume --bams bam_paths.txt biostars9540675.nf

[LDT]

That's a nice suggestion, Pierre. I appreciate. I am a bit behind in next flow but your post boosts me to go in that direction even faster