Issue with Cluster

Question

[aortigas]

Hello everyone,

I am trying to run a program called HPVDetector inside the Compute Canada cluster which uses SLURM to schedule jobs. For some reason, when I simply execute the program inside the head node it works perfectly, but when I submit it as a job with sbatch or srun, it takes MUCH longer (not counting queue waiting times), and the output files are all empty (even though the program shows no errors throughout its run). I have looked absolutely everywhere for a solution or explanation but I can find none. I am new to HPC and would really appreciate any help. Thank you so much in advance.

[steve]

Might want to familiarize yourself with this: https://slurm.schedmd.com/sbatch.html

[aortigas]

Thank you very much! This looks extremely useful and complete. Will definitely take a look.

Answer 1

[Manoj]

For time duration :

Could you please check if memory and partition of your script look similar like this

SBATCH --mem=16G

Above 16G means 16 Gb memory have been allocated but if your program demand more memory replace 16G with MaxMemPerNode

SBATCH --partition=standard

Here generally we use standard but you may replace standard with fast

Note: As other people must be using that cluster at the same time, use these two parameters wisely

Regarding output file

I think there might be problem with some variable or file name. Take special care of $ symbol

Hope this helps :)

[aortigas]

You sir are an absolute genius. I just did what you said about the memory:

SBATCH --mem=MaxMemPerNode

And it ran perfectly, actually even BETTER because it was done in under a minute. I cannot thank you enough. Thank you so much!!!