How to configure Slurm for local Galaxy bioinformatics analysis server.

Question

[cbass]

Hi, I am wondering how to exactly set the usage of CPU and RAM with Slurm, since this is my first time using Slurm (or any workload manager/job scheduler for that matter). Can I just create new partitions and set these partitions for the destination id together with slurm as the runner? And what is the param id? I had the following settings in mind for Slurm, but am wondering if this would actually work or not. Also, I am using this for my private galaxy server (https://galaxyproject.org/) in order to run tools more efficiently, in case this might be important. Any feedback/help is appreciated! Thanks in advance!

Here are the parts of my files with the settings I had in mind:

galaxyservers.yml:

# Slurm
slurm_roles: ['controller', 'exec']
slurm_nodes:
- name: localhost 
  CPUs: 14 # Host has 16 cores total
  RealMemory: 110000 # 110000MB, viewed with command 'free --mega' which shows 135049MB total free
  ThreadsPerCore: 1
slurm_partitions:
  - name: main
    Nodes: localhost
    Default: YES
slurm_config:
  SlurmdParameters: config_overrides   # Ignore errors if the host actually has cores != 2
  SelectType: select/cons_res
  SelectTypeParameters: CR_CPU_Memory  # Allocate individual cores/memory instead of entire node

job_conf.xml.j1:

<job_conf>
    <plugins workers="14"> #cores
        <plugin id="local_plugin" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner"/>
        <plugin id="slurm" type="runner" load="galaxy.jobs.runners.slurm:SlurmJobRunner"/>
    </plugins>
    <destinations default="slurm">
        <destination id="local_destination" runner="local_plugin"/>
        <destination id="partition_1" runner="slurm">
            <param id="p01">--nodes=1 --ntasks=1 --cpus-per-task=1 --mem=4000</param>
            <param id="tmp_dir">True</param>
        </destination>
        <destination id="partition_2" runner="slurm">
            <param id="p02">--nodes=1 --ntasks=1 --cpus-per-task=2 --mem=6000</param>
            <param id="tmp_dir">True</param>
        </destination>
        <destination id="partition_3" runner="slurm">
            <param id="p03">--nodes=1 --ntasks=1 --cpus-per-task=4 --mem=16000</param>
            <param id="tmp_dir">True</param>
        </destination>
    </destinations>
    </limits>
     <tools>
        <tool id="tool_example_1" destination="partition_1"/>
        <tool id="tool_example_2" destination="partition_2"/>
        <tool id="tool_example_3" destination="partition_3"/>
     </tools>
</job_conf>

[Mensur Dlakic]

This is about resource management, not about bioinformatics.

[cbass]

True, but it was recommended to me to post the question on this forum, since it is related to the Galaxy project, which is a bioinformatics platform, and I could possibly find people on here who also use this platform for bioinformatic analysis who know how to fix this problem, therefore the question was posted on here.

[Pierre Lindenbaum]

just curious: what is the point of managing a "private" instance of galaxy ?

[Jean-Karim Heriche]

Access to non public data on your institution or group own storage and compute.

[Pierre Lindenbaum]

well, managing galaxy is part of the bioinformatician job.

Answer 1

[Jean-Karim Heriche]

Try asking on the Galaxy Help forum: https://help.galaxyproject.org/ or on gitter: https://gitter.im/galaxyproject/Lobby