How to let WGCNA use multiple cores
1
1
Entering edit mode
7.6 years ago
syrttgump ▴ 50

Hi guys, I am using WGCNA to deal with some large datasets (~20000 genes on ~1000 samples). I read in the help documents that WGCNA support multi-thread to speed up calculation. But when I run functions like blockwiseModules, TOMsimilarityFromExpr, cor etc, I can set parameter nThreads, it works. But when I check CPU usages by "top", it seems all multiple threads are running on one CPU core, so I didn't gain any advantage by multi-thread (I am using a linux cluster with 24 CPU cores, and set enableWGCNAThreads() already). How can I set WGCNA to use multiple cores?

WGCNA multe-thread parallel • 7.7k views
ADD COMMENT
0
Entering edit mode

have you fix it problem?

ADD REPLY
0
Entering edit mode
6.3 years ago

Can you try running these lines at the top of your script(s):

require(parallel)
require(doParallel)

cpucores <- makeCluster(detectCores(), type='PSOCK') ; 

registerDoParallel(cpucores) ; 

Sys.setenv("MC_CORES"=cpucores) ;

The last line (setting MC_CORES variable) is only for Windows OS.

Also take a look at these other WGCNA functions: https://rdrr.io/cran/WGCNA/man/allowWGCNAThreads.html

ADD COMMENT
0
Entering edit mode

I met this problem too, and using the above code could not solve the problem.

Aifu

ADD REPLY
0
Entering edit mode

Then please show what is the error message that you are receiving. In certain situations, you will, simply, not have enough power on your computer to do the analysis that you want. You should consider reducing the size of your input data [to WGCNA]

ADD REPLY
0
Entering edit mode

I haven't met any error, the WGCNA works well, but it only use one CPU, I've tried using enableWGCNAThreads() or allowWGCNAThreads(nThreads = 22) and so on, but have no effect. I think maybe this package just use one CPU. And I use small dataset or large dataset to WGCNA, also no change.

ADD REPLY
0
Entering edit mode

Which operating system are you using? The parallel functions in R differ based on the 'system architecture' that you use.

ADD REPLY
0
Entering edit mode

Maybe, I use Ubuntu 16.04 LTS, and the R package is 3.5.0 version. Can the WGCNA(r package) works with multiple threads on your system?

ADD REPLY
1
Entering edit mode

I also use Ubuntu 16.04.

Firstly, try the enableWGCNAThreads() function, and do not use allowWGCNAThreads()

If that does not work, then first run these lines:

#Set CPU cores for parallel-related functions
  cpucores <- 16
  require(parallel)
  options("mc.cores"=cpucores)

  #Set CPU cores for doParallel-related functions
  require(doParallel)
  cores <- makeCluster(detectCores(), type='PSOCK')
  registerDoParallel(cores)

...then run enableWGCNAThreads() again.

Check if multiple threads are running via the top command in BASH. You will see the R command being forked

ADD REPLY
0
Entering edit mode

Thanks Kevin, I try the codes above and add enableWGCNAThreads(), but this could not increase the threads number. The program has cost more than 10 hs, and is still running for just one thread. One of my friends told me that WGCNA just uses one core. Maybe there're other methods to increase the thread number. I upload the data and paste my codes as follows:

library(WGCNA)
options(stringsAsFactors = FALSE)
enableWGCNAThreads(10)
lnames=load(file="filtered_data_ready_for_WGCNA_0.5tpm.RData")
datExpr <- datExpr0
rm(datExpr0)
setwd("./maxP0.05")
power <- 7
bwnet = blockwiseModules(datExpr, maxBlockSize = 46000, power = power, networkType = "signed hybrid", TOMType = "unsigned", minModuleSize = 30, corType="bicor", corOptions = "use = 'p', maxPOutliers = 0.05", reassignThreshold = 0, mergeCutHeight = 0.15, numericLabels = TRUE, saveTOMs = TRUE, saveTOMFileBase = "Signed_hybrid_TOM-blockwise_maxP0.05", verbose = 3)

You can try this, if you could run with multiple threads please share the good news with me!

data link: https://de.cyverse.org/dl/d/1D0B60A0-D1EF-4A4F-A13E-A21901301900/filtered_data_ready_for_WGCNA_0.5tpm.RData

ADD REPLY
0
Entering edit mode

Okay, let me try

ADD REPLY
1
Entering edit mode

I tried, and, indeed, that particular function does not have parallel-processing enabled - I looked through the code. I then found this, where one of the WGCNA developers states exactly this:

I notice a lot of for loops in the blockwiseModules code, which could be replaced with foreach %dopar%. I initially tried but ran into errors because the function is >3000 lines of code.

I previously 'parallelised' another much smaller functiun that also performs k-means (or other) clustering: https://github.com/kevinblighe/clusGapKB

ADD REPLY
0
Entering edit mode

Thank you very much Kevin for your detailed reply! I accept the reality that I should use just one thread to run WGCNA at the moment.

ADD REPLY
0
Entering edit mode

im not sure about it. this is what my initial code for wgcna

library(WGCNA)
enableWGCNAThreads()

it works ...

ADD REPLY
0
Entering edit mode

Do you mean that many threads work simultaneously? Could you please show your system information, and WGCNA version?

I tried again and failed again...

ADD REPLY
1
Entering edit mode

krushnach80, good to see you again. Can you confirm that it is the blockwiseModules() function that you are using? We believe, based on This, that it is just the blockwiseModules() function that is not yet multi-thread enabled.

I imagine that you are using a different WGCNA function.

ADD REPLY
1
Entering edit mode

@Kevin sorry for the late response yes im not using "blockwiseModules()" .

ADD REPLY

Login before adding your answer.

Traffic: 2482 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6