Snakemake: set cluster cores per rule

Question

[samesense]

I need to download hundreds of large files, and run each of them through my snakemake pipeline. The file download is fast compared to my downstream pipeline. I'd like to limit the number of parallel downloads to 5, but allow the downsteam processing to use 100 cores. In snakemake, is there a way to limit the number of cores used by a certain rule? I picture 5 cores constantly grabbing data, while my other cores are working on the data I've already downloaded. If I run snakemake as usual with 100 cores, it will try to download all files at once, and overload the server.

[samesense]

I tried setting a download resource, according to this answer

This did not limit the downloads as expected.

Answer 1

[maciek.kandula]

Is this what you're looking for: https://bitbucket.org/johanneskoester/snakemake/wiki/Documentation#markdown-header-threads?

Otherwise, you can try posting in the official Snakemake Google group: https://groups.google.com/forum/#!forum/Snakemake The author is pretty responsive.

[samesense]

I saw that too. I think it would multithread my download, but wait on the rule to finish before moving onto the downstream pipeline. I'll ask the user group.