How to let WGCNA use multiple cores

Question

[syrttgump]

Hi guys, I am using WGCNA to deal with some large datasets (~20000 genes on ~1000 samples). I read in the help documents that WGCNA support multi-thread to speed up calculation. But when I run functions like blockwiseModules, TOMsimilarityFromExpr, cor etc, I can set parameter nThreads, it works. But when I check CPU usages by "top", it seems all multiple threads are running on one CPU core, so I didn't gain any advantage by multi-thread (I am using a linux cluster with 24 CPU cores, and set enableWGCNAThreads() already). How can I set WGCNA to use multiple cores?

[yplee614]

have you fix it problem?

Answer 1

[Kevin Blighe]

Can you try running these lines at the top of your script(s):

require(parallel)
require(doParallel)

cpucores <- makeCluster(detectCores(), type='PSOCK') ; 

registerDoParallel(cpucores) ; 

Sys.setenv("MC_CORES"=cpucores) ;

The last line (setting MC_CORES variable) is only for Windows OS.

Also take a look at these other WGCNA functions: https://rdrr.io/cran/WGCNA/man/allowWGCNAThreads.html

[afli]

I met this problem too, and using the above code could not solve the problem.

Aifu

[Kevin Blighe]

Then please show what is the error message that you are receiving. In certain situations, you will, simply, not have enough power on your computer to do the analysis that you want. You should consider reducing the size of your input data [to WGCNA]

[afli]

I haven't met any error, the WGCNA works well, but it only use one CPU, I've tried using enableWGCNAThreads() or allowWGCNAThreads(nThreads = 22) and so on, but have no effect. I think maybe this package just use one CPU. And I use small dataset or large dataset to WGCNA, also no change.

[Kevin Blighe]

Which operating system are you using? The parallel functions in R differ based on the 'system architecture' that you use.

[afli]

Maybe, I use Ubuntu 16.04 LTS, and the R package is 3.5.0 version. Can the WGCNA(r package) works with multiple threads on your system?

[Kevin Blighe]

I also use Ubuntu 16.04.

Firstly, try the enableWGCNAThreads() function, and do not use allowWGCNAThreads()

If that does not work, then first run these lines:

#Set CPU cores for parallel-related functions
  cpucores <- 16
  require(parallel)
  options("mc.cores"=cpucores)

  #Set CPU cores for doParallel-related functions
  require(doParallel)
  cores <- makeCluster(detectCores(), type='PSOCK')
  registerDoParallel(cores)

...then run enableWGCNAThreads() again.

Check if multiple threads are running via the top command in BASH. You will see the R command being forked

[afli]

Thanks Kevin, I try the codes above and add enableWGCNAThreads(), but this could not increase the threads number. The program has cost more than 10 hs, and is still running for just one thread. One of my friends told me that WGCNA just uses one core. Maybe there're other methods to increase the thread number. I upload the data and paste my codes as follows:

library(WGCNA)
options(stringsAsFactors = FALSE)
enableWGCNAThreads(10)
lnames=load(file="filtered_data_ready_for_WGCNA_0.5tpm.RData")
datExpr <- datExpr0
rm(datExpr0)
setwd("./maxP0.05")
power <- 7
bwnet = blockwiseModules(datExpr, maxBlockSize = 46000, power = power, networkType = "signed hybrid", TOMType = "unsigned", minModuleSize = 30, corType="bicor", corOptions = "use = 'p', maxPOutliers = 0.05", reassignThreshold = 0, mergeCutHeight = 0.15, numericLabels = TRUE, saveTOMs = TRUE, saveTOMFileBase = "Signed_hybrid_TOM-blockwise_maxP0.05", verbose = 3)

You can try this, if you could run with multiple threads please share the good news with me!

data link: https://de.cyverse.org/dl/d/1D0B60A0-D1EF-4A4F-A13E-A21901301900/filtered_data_ready_for_WGCNA_0.5tpm.RData

[Kevin Blighe]

Okay, let me try

[Kevin Blighe]

I tried, and, indeed, that particular function does not have parallel-processing enabled - I looked through the code. I then found this, where one of the WGCNA developers states exactly this:

• https://support.bioconductor.org/p/86147/#86151

I notice a lot of for loops in the blockwiseModules code, which could be replaced with foreach %dopar%. I initially tried but ran into errors because the function is >3000 lines of code.

I previously 'parallelised' another much smaller functiun that also performs k-means (or other) clustering: https://github.com/kevinblighe/clusGapKB

[afli]

Thank you very much Kevin for your detailed reply! I accept the reality that I should use just one thread to run WGCNA at the moment.

[1769mkc]

im not sure about it. this is what my initial code for wgcna

library(WGCNA)
enableWGCNAThreads()

it works ...

[afli]

Do you mean that many threads work simultaneously? Could you please show your system information, and WGCNA version?

I tried again and failed again...

[Kevin Blighe]

krushnach80, good to see you again. Can you confirm that it is the blockwiseModules() function that you are using? We believe, based on This, that it is just the blockwiseModules() function that is not yet multi-thread enabled.

I imagine that you are using a different WGCNA function.

[1769mkc]

@Kevin sorry for the late response yes im not using "blockwiseModules()" .