So far, I can describe my question only in general words, but hopefully someone can help me further narrow it down. I am trying to run modified version of GATK pipeline through cromwell in a server mode on a slurm cluster.
So far I setup a slurm cluster with one node, as a test environment and everything works reasonably well: I'm able to push several instances of my pipeline in parallel to the cromwell server and slurm creates a nice queue of tasks. The problem is, that currently I am hitting the limit of IO operations on the disk where cromwell server is running, which slows everything down:
Here is the result of 18 pipelines submission with 1h interval. As can be seen I/O operations are at it's limit restricting the overall performance.
Now I'd like to add some more nodes to Slum to deal with it. _BUT_ from what I learn so far - all slurm nodes will use the same shared disk, meaning that I will not gain any performance increase from additional computational resources. Is it correct? Or can I force slurm to use local hard drives for each node? If so, how I can configure cromwell server and nodes to correctly copy files back and force?
The question is quite vague but I thank the main idea is clear - ho to increase performance over what I have by addition more nodes to the cluster (if it is possible at all)
Best wishes, Eugene
are you using a Lustre file system?
Hi, sorry for slow response, I was educating myself a bit on the topic) No, currently we do not use any special fs, but seems that it is unavoidable, thanks! What is your experince with lustre?
It is essentially abouth the same issue described here https://github.com/broadinstitute/cromwell/issues/5802 also witout any replies :(