Question

How to use whole computer cluster power for computation of a single job?

0

Entering edit mode

10.7 years ago

sammy.ich17 ▴ 20

I know the computations by breaking the files and parallelize it in SGE or SLURM. I want to use whole computation power (20 node , 80 cores) for a single job without breaking it. If I just submit it (a single perl script for large computation by AutoFact program) from master node it is occupying the cluster but not computing with 20 node and 80 cores.

blast perl sequence alignment • 2.1k views

ADD COMMENT • link updated 10.7 years ago by Alex Reynolds 36k • written 10.7 years ago by sammy.ich17 ▴ 20

1

Entering edit mode

Hi,

I have a feeling more details (specifics on what you're trying to accomplish, what the perl script does, etc) will help you get an answer that is best suited to your scenario. The question in its current form is pretty vague and offers little detail on your *exact* current procedure. Pl help us help you better :)

ADD REPLY • link 10.7 years ago by Ram 44k

2

Entering edit mode

"for a single job without breaking it" seems incompatible with "computing with 20 node and 80 cores."

ADD REPLY • link 10.7 years ago by Pierre Lindenbaum 164k

1

Entering edit mode

My question though is what is the "it" here? Breaking the input file or breaking as in damaging the process? OP needs to be more specific and unambiguous IMHO.

ADD REPLY • link 10.7 years ago by Ram 44k

score 3 · Answer 1 · 2014-04-25

3

Entering edit mode

10.7 years ago

Devon Ryan 105k

You'd have to rewrite your perl script to use MPI so that you can dole out computations between the nodes. Just running a program on a cluster doesn't make it parallel, you have to design it to use the resources.

ADD COMMENT • link 10.7 years ago by Devon Ryan 105k

score 2 · Answer 2 · 2014-04-25

Looking at the AutoFACT scripts, it's not clear that MPI will help. Looking over the methodology, it looks like it just does a bunch of searches to various databases, correct? If it's not a computationally intensive process for AutoFACT, that suggests a parallelization approach, where each of various searches can be set on their own node. Take a look at the Schedule::SGE module and consider modifying the AutoFACT code to support scheduling jobs, where jobs are database searches.