How to allocate memory and CPU without SLURM in a linux cluster?

Question

[anikcropscience]

Hello,

I am running the following script in a Linux-based cluster.

for FILE in $FASTQ_DIR/*.fastq
do
    BASENAME=$(basename $FILE .fastq)
    bwa mem -t 4 -M $REFERENCE $FILE > $BAM_DIR/$BASENAME.sam
done

When I check the status after running this code, it shows that I am using only 1 CPU and 0.4% of the memory. The cluster has 256 CPUs and 512 Gb RAM.

Is there a way to specify more number of CPUs and RAM to be used for specific codes? I do not have the SLURM system installed in the cluster.

Previously I used the following code, but it did not run or do anything.

systemd-run --scope -p CPUQuota=60% -p MemoryMax=300M --user ./test.bash

Thank you

[GenoMax]

The cluster has 256 CPUs and 512 Mb RAM

You must mean 512 GB of RAM.

If you are working on a cluster then presumably it has some kind of job scheduling system. This question is best addressed to your local systems tech support.

[anikcropscience]

Ah sorry. Yes, it is 512 GB of RAM. I addressed this and got a reply that they are working on building a job scheduling system. So at the moment, there is no such scheduling option on this cluster.

[GenoMax]

Looks like you already know how to limit system resources (systemd-run) based on your original post.

Please keep in mind that aligners may not use all cores/threads at all times. As long as you are specifying the right options things should work.

[Nicolas Rosewick]

read the doc :https://blog.ronin.cloud/slurm-intro/

[anikcropscience]

There is no SLURM in the cluster I am working on.

[Ram]

Oops, my bad: I saw the keyword slurm on OP's title and added it as a tag. OP mentions "without SLURM"

[Ram]

You deleted a bunch of posts that had all received feedback. Please do not do this again, or your account will be suspended.

Answer 1

[i.sudbery]

By specifying -t 4 in the bwa command line, you are telling bwa to use 4 threads. Now, each of those threads will not neccessary max out a CPU, because at any one time, each thread might be waiting for data from disk, or waiting for another thread to finish what it is doing to provide it with the information it needs.

If you want bwa to use more CPU at once, you need to increase the number of threads it is allocated by using a higher number than 4 in your -t option. However, be aware that generally there is a point of dimishing returns on allocating more an more threads to a single task, where adding more threads doesn't help becuase you are bottlenecked by something else.

bwa will use the memory it uses - in the case of batch linux apps like this, allocating more or less memory makes no difference to speed as long as memory is sufficient that there is no swapping to disk (large linux servers usually have disk swapping disabled anyway).

However, the best way to speed this up would be to run multiple instances of bwa at once. At the moment you are running on one fastq, waiting for that to finish, and then running on the next. You would improve things by running 4 or 5 instances at once. You can do this with the parallel tool https://www.gnu.org/software/parallel/.

[anikcropscience]

Thank you for the reply.

Answer 2

[Mensur Dlakic]

Presumably your system has 512 Gb of memory, because 512 Mb would be tiny. So if you are specifying 300 Mb on command line, that would be about 0.4% of memory.

[anikcropscience]

No, sorry. It is 512 Gb of memory.

[Ram]

GB, not Gb. They mean different things.

[Ram]

Mensur, it should be GB, not Gb. Though Gb is not exactly fitting to this context, it is still relevant and we should be clear when correcting others.

[Mensur Dlakic]

Sure thing - I appreciate the pedantry as much as the next person.

Speaking of things we should be doing, I suggest that we should be professional to strangers by not telling them they are unprofessional.

[Ram]

I appreciate the feedback. I'll keep that in mind.