Hi,
I have 4 directories with 4 fastq files each. I need to run fastqc
on them at the same time. I use :
sbatch -p node --mem=32G --ntasks 16 --cpus-per-task=8 --wrap='find . -type d -name data -prune -o -name '*.fastq.gz' -print | parallel -j 16 fastqc -t 8 -o {//}/ {}'
I normally have 16 parallel tasks because of the 16 fastq files , and I want to allocate 8 cpus / task (I do have enough ressources to do it). Is that work properly ? Because when I look at the top
command, I see my 16 "java" jobs with the %CPU value blocked at 100%. I should have 800% each , because of the 8 cpus / task no?
Best
make your life easier, use a workflow manager. https://vibbits-nextflow-workshop.readthedocs.io/en/latest/nextflow/first_pipeline.html
You're right, I'll try that later. But if you could unlock my
sbatch
issue, it would be fine.sbatch
individual FastQC jobs via afor
loop. You have a bonafide job scheduler (SLURM) available. There is no need to useparallel
.