Is a simple Wilcoxon rank sum test good enough to identify differentially expressed genes?

Question

[suvratha]

Hi, I've noticed that the Wilcoxon rank-sum test is used by many R packages to calculate differential expression. Is this a good test to estimate that when you compare it to other packages like DESeq2 and edgeR which calculate differential expression with many more factors involved and would that not be better than just a Wilcoxon test?

Answer 1

[Michael Love]

I agree with Devon here except I'd put the number even lower than 100 for trusting Wilcoxon. Rank based tests will find differences with sufficient power once you have, say dozens of cells with non-zero counts in each group that you want to compare.

We used Wilcoxon tests with Gibbs samples to also account for uncertainty from multimapping reads in Swish:

https://academic.oup.com/nar/article/47/18/e105/5542870

Jump to "scRNA-seq simulation" and "scRNA-seq of mouse neurons" for an examination of how Wilcoxon can be used effectively for comparing moderate-sized groups of cells (e.g. ~20 or higher) even in the presence of uncertainty from multi-mapping reads.

[suvratha]

So do you mean that one should trust Wilcoxon if the number of cells in question is less (eg - <100) and if the number cells are higher (>100 or more), it is better to trust DESeq2 instead?

Answer 2

[Devon Ryan]

Have you looked at any of the comparison papers on scRNA-seq differential comparisons? Given the number of cells involved in clusters a Wilcoxon test tends to perform quite well given its simplicity.

[suvratha]

which would you trust better, the results obtained from the Wilcoxon test or from DESeq2 which obviously differ?

[Devon Ryan]

It will depend on how many cells are in the clusters in question. If they're lowish (by single-cell standards, so maybe <100 cells) I'd definitely trust DESeq2 more, otherwise rank-based methods should generally be competitive (until relatively recently DESeq2 was quite slow when dealing with larger cell numbers, though this has been fixed in newer releases). In general, if you have a lot of cells and the two methods disagree strongly then I would suggest looking at the underlying count distributions and determining why that is. The reason will tell you whether you want to include those genes or not downstream.

[suvratha]

the number of cells I've is about 1500. the thing is that, when I'm looking at my gene of interest from both the analysis from the Wilcoxon test and DESeq2, my gene gets upregulated for one population of cells from the Wilcoxon test and the same gene is reported as downregulated for the same population of cells using DESeq2. this is why I'm having a hard time trusting the statistic tests in both the methods.

[suvratha]

I'm using 2 different tools for this, the Wilcoxon test is being used by the Cytosplore tool and I'm using DESeq2 in Seurat to get my differentially expressed gene list. I know I can't expect the exact same results between the two tools but can it be so drastic change in the expression of my gene of interest?

[jared.andrews07]

Not sure, but the default test for Seurat is the Wilcoxon test, so maybe use that to compare to your DESeq2 results? At least then you'll remove any differences due simply to the tools used. I'm unfamiliar with Cytosplore, is it using the same clusters as your Seurat analysis?

[Devon Ryan]

I agree 100% with Jared, stick with doing everything within Seurat for consistency (otherwise you'll inevitably end up with an apples and orange comparison).

[suvratha]

I did that already, I did compare the Wilcoxon test results from Seurat to the results from Cytosplore and the gene of my interest does not show up at all as a differentially expressed gene in Seurat at all. This was the reason I decided to go for the DESeq2 method which I thought was taking into consideration a lot more factors while calculating differential expression. And hence the question is it good enough to trust the results of just the Wilcoxon test while doing differential expression testing.

[jared.andrews07]

Okay, but what about the gene sets as a whole? Is there 95% overlap between the two methods in Seurat? What's the overlap between Cytosplore and Seurat both using the Wilcoxon test?

If you're changing statistical tests just to get a significant p-value, you're performing "p-value hunting", which is best avoided. This study is a very thorough review of differential expression methods in scRNA-seq, which shows Wilcoxon tests are very robust, scale well, and provide good recall/precision (see Figure 5).

[suvratha]

There is hardly a 10-20% overlap between Cytosplore and Seurat both using the Wilcoxon test.

[jared.andrews07]

Cytosplore is meant for CyTOF. Why are you trying to use it on scRNA-seq data? I would stick with Seurat (or any other well-regarded scRNA framework).

[suvratha]

Cytosplore Viewer is what I'm using and it is designed to handle single-cell RNA-seq data as well.