Fpkm Comparison Using T-Test Or Mann-Whitney U Test

Question

[Vikas Bansal]

Dear all,

I was working through RNA-seq data set from two stages (before differentiation and after differentiation) of cell line. I want to check for 585 genes (these genes are interesting for us) if expression is significantly higher after differentiation or not. I used t-test as well as Mann-Whitney U test (Wilcoxon rank-sum test) on FPKM values of these genes. The p values are -

0.14 (t-test)
8.1e-05 (MWW)

Questions

Which test should I use and why?

Is it normal to see such a difference in p values using t-test and MWW?

P.S: I know the assumptions behind the tests but I do not know if FPKM values are normally distributed or not.

EDIT: Below are the FPKM values of my 585 interesting genes:

FPKM_undiff    FPKM_diff
0    0.00979993
0    0
0.0156791    0.00761548
0.0543874    0.129448
0    0
0    0
0.00650977    0
0    0.00981821
0.0082179    0
0    0
0    0.0275734
0    0
0    0
0.0506587    0.058278
0    0
0    0
0    0
0.0527133    0.0320534
0    0
0    0
0.0141636    0
0    0
0    0
0    0
0    0
0    0.021557
0.476441    0.74297
0    0.00216964
0    0
0    0.00171133
0.0711397    0.0782215
0    0
0    0
0    0
0    0
0    0
0    0
0.0129147    0.0265309
0    0
0    0
0.592774    2.19515
0    0
0    0
0    0
0    0
0.0029552    0.00374453
0    0
0    0
0    0
0    0.0281069
0    0
0    0
0    0
0    0.00266207
0    0
0.461615    0.160647
0    0
0    0
0    0.146067
0    0
0    0.0145832
0.116135    0.239599
5.1295    0.790805
0    0.00466653
0.0557738    0.163162
0.13616    0.168267
0.0515826    0.0217485
0    0
0.112333    0.0306879
0    0
0    0
0    0.0598548
0    0
0    0.00341391
0.0112512    0
0.011781    0.61845
0    0
0.00536391    0.239729
0.176625    0.346842
0    0
0.0409464    0.0518428
0    0
1.40836    1.88935
0    0
0.188303    0.821979
0    0.00674785
0    0
0    0.00221935
0.0470863    0.784393
0    0
0.0101794    0.00131401
0.025838    0.00624204
0.255868    0.192638
0.0138707    0.464247
0    0
0    0
0.213469    0.348231
0.0200612    12.1792
0    0
0.00560411    0.00690147
0.059252    0.0131399
0.44933    0.0823733
0    0
0.00920092    0.126886
0    0.0037034
0.0227527    0.0194174
0.0351402    0.0159705
0.215587    0.285912
0.00876542    0
2.42575    0.0361371
0    0.0972781
0    0
0    0
0.0204054    0.0103906
0    0.0189586
7.36312    12.6552
0    0
0.0327914    0.0167844
0.986454    17.8484
0.00113704    0.00612809
0    0
0.0134198    0.058368
0    0
0.200898    0.258552
0.0199969    0.0553026
0    0
0.167788    0.261856
0    0
0.0360742    1.08595
0    0.0136952
0.0223617    0.0473491
0    0
0    0
0.286363    3.29673
0    0.00604735
0    0
0    0
0    0
0.00245598    0.00656835
0.0832634    0.053754
0.00563902    0.00457595
0    0
0.00360985    0.00919395
0.526873    0.730862
0    0.0140973
0    0
0.0181377    0.179448
0    0
0    0.00564234
0.00314785    0.00195837
0    0.0104087
0    0.00234022
0    0
0.093442    0.00967311
0.102117    0.00501067
0    0
0.177181    6.46037
0    0.00332104
0    0
0.563838    0.0788021
0    0.0938891
0.257809    36.606
0.0134618    0.00168805
0.00327106    0
0.00386116    0.0191951
0.015911    0.2473
0    0
0    0
0.0159217    0.110809
0    0.00610845
0    0
0    0
0    0.0149144
0.0095157    0.033828
2.82654    2.90175
0.567591    0.173613
0.133945    0.390562
0.0268004    0.273638
0.0504087    1.11836
0    0
0.00709921    0.0171702
0    0
0.00338803    0
0.695089    0.755439
0    0.0368949
0    0.0398636
0.00910244    0.0114124
0    0
0    0.392474
0    0
0.00210047    0.00520475
0    0.00232866
2.29881    1.16059
0.553412    0.093849
0.0120985    0.217203
0.00173353    0.015531
0    0
0    0
0.00709787    0
0    0
35.4183    33.2559
0.0203704    0.0301657
0.0162436    0.382246
0.0814903    0.431288
0    0
0.854938    0.269161
0.279646    0.269814
0.0132792    0.0127739
0    0.00405924
0.433035    0.834682
0    0
0.127141    0.165564
0.00406106    2.79114
44.3045    37.3835
0.368794    0.800279
0    0.0107781
0.00246019    0
0    0.0647367
0.122162    0.0350259
0.00698102    0
0.0235323    0.0573162
0    0
0.553894    263.541
0.0275198    0.0110146
0    0
0.0135625    0.00461561
0.245475    0.234519
0.0482715    0.0783393
0    0.119063
0    0.00228496
0    0.00591806
0    0.240085
0.138895    0.479992
0    0
0.452653    0.298752
0.00089268    0.0123278
0.00664013    1.25859
0.0137128    0.0706789
0.107589    0.0148237
0    0.00244057
0    0
0    0
0    0
0.00884167    0
0    0
0    0
0    0
0    0
0    0.0465225
0    0
0    0
0.0127877    0.00759114
0    0
0    0
0.0143108    0.0290303
0.0048602    0.00625753
0    0
0    0
0    0
0    0
0.0378325    0
0    0
0.11311    0.0408646
0.0208299    0.0238735
0.00607765    0.088116
0.408866    0.364383
0.368514    0.00288614
0.0475567    0.608063
3.44219    1.36913
0.00347198    0.00409186
0.0999656    0.0372491
0.040979    0
0.00679074    0.00499061
0.0154534    0
0.0204074    0.0422706
47.0831    47.8712
0    0
0.00143883    0
0.172199    0.376105
0.0765674    0.200574
0    0.0039195
0.209268    78.8024
0.0228858    0.108583
0    0
0.005481    0.00224165
0.0451118    0.0906822
0.101011    1.81411
0.0155015    0.00612381
0    0
0    0
0.0104911    0.0664517
0.0049624    0.00603241
0.00279289    0.0147797
0.129573    0.0559773
4.26291    8.18277
0.00522598    0.0764129
0    0
0    0.00968798
0    0.333089
0    0
0.23777    0.279177
0.192399    0.367337
0.0059381    0.147093
0.0309991    0.0182781
0.00569411    0.00362466
0    0.1186
0.00621093    0
0.00603825    0
0    0
0    0.0154021
0    0
0.00628578    0.0869839
0    0.0313295
0.00126726    0
0    0
0    1.01982
0.0016646    0.00808306
1.49054    0.728432
0.142694    0.833148
0    0
0.0621361    0.10469
0    0
0    0
0.0663542    0.0534874
0.00541785    0.0265786
0.582843    0.126975
0    0
0    0
0    0.189247
0.0328527    0.0162086
0    0
0    0
0    0
7.89499    4.44444
0.00461955    0
0    0
0.0130804    0.93429
0    0.00416887
0.00356088    0
0.0137056    0.0665561
0.0026154    0
0.0260207    0.0953219
0.0880793    0.0107424
0    0
0    0
0    0
0    0
0.25522    0.290681
0.0788486    0.00761706
0    0.0543497
0    0
9.12539e-06    0.0104041
0.0144905    0.00751801
0    0.00294302
0.147805    0.0788936
0    0
0    0
0    0
0    0
0    0
0    0
0.0557738    0.163162
0    0
0.338455    0.0837984
0.0157758    0
0.257124    0.211645
8.25195    38.014
0.456382    0.934556
0.0368361    0.243147
0.284639    0.325576
0    0.00541464
0.00119835    0
0    0.127273
0.30078    0.118195
0    0.0139656
0.00144155    11.6424
0    0
0.335903    473.609
0    1.77205
0    0
0.00866648    0.108523
0    0.0261056
0.0130487    0.022707
5.01997    5.64537
1.21398    0.0723739
0.0224691    0.0315844
0    0
0    0
0    0.012496
0    0.00226908
0    2.70307
0.0928407    0.191784
0    0.00596784
0    0
0.00286748    0.0523854
0    0
0    0
0    0
0.0131631    6.16212
0    0.0239042
0    0.00508183
0    1.20761
0    0
0    0
0    0
148.358    149.648
148.358    149.648
148.358    149.648
148.358    149.648
148.358    149.648
148.358    149.648
148.358    149.648
148.358    149.648
148.358    149.648
148.358    149.648
148.358    149.648
148.358    149.648
148.358    149.648
148.358    149.648
0.240246    0.565728
0.335682    0.582158
0.0615637    0.121294
0.929209    2.11455
0.15134    0.0146172
0    0
0    0
0.00442694    0.00511991
0.0213365    0.00971155
0    0.267421
1.5106    3.01687
0.00566643    0.019987
0    0.0297383
2.03318    184.926
0.0141299    0.0169819
0.0785617    0.0235894
0.00904319    0.0242694
0    0
0    0.00636067
0.405022    0.763255
0    0
0.153281    0.342909
1.39708    1.66659
0    0.00453808
0.00501467    0
1.30329    0.626499
0    0
1.58934    2.5728
0.319493    0.228329
0.0426172    0.0312349
0    0
0.0259343    0.108931
0.287356    0.257748
0.0412072    1.49625
13.8724    7.79341
5.14581    0.624423
0.019759    0.0493204
5.02407    10.5572
0.0116082    0.104398
0    0
0.766728    0.123647
0    0
0.00406371    0
0.016961    5.35044
0    0.00240722
0    0.0123523
0.178006    28.1923
0.00956514    0.00577151
0    0.0950938
0.0262632    0.493054
0.0366789    2.30324
0    0
0.0272311    0.11721
0    0.0076969
0    0.00214225
0    0.0681147
0.0529286    1.22047
0    0
0    0
0    0
0    0
0    0.0407487
1.63428    2.24584
0    0
0.0141748    0.0180332
0.0108742    0.00545133
0    0
0.205214    0.279312
0    0
0    0.00356799
0    0.0449568
0.00551323    0.0304052
0.0566858    0.399034
3.64091    0.080254
0.0839108    0.016863
0.0183947    0.0515463
0    0.00321718
4.03906    9.39997
0    0.00142553
0    0
0.170233    3.4909
0    0.0209324
0    0.0298148
0.00533474    0.00226183
0    0.0119659
0    0
0.0854973    0.0886838
0    0.047859
0.00769751    0.0288787
0.12285    0.0170485
5.52833e-05    7.17087e-05
0.0118421    0
0    0
0.0100909    0.0222188
0.00700353    0.00223443
0    0.0241404
0    0
0.00942    0.00180392
0    0.00380445
0    0
0.00786214    0.00325225
0.0565028    30.6016
0.955844    0.850619
0    0
0.0014712    0.00461979
0    0
0    0
0.00897945    0.0304897
0.223273    0.0783447
0    0
0    0
0.0035911    0.908899
0    0
0.0017013    0.00225269
0    0.0194741
0    0.098015
0.0691567    0.140123
0    0
0.935291    2.34976
0.0120628    0.0101693
0.0066743    0.804577
0    0
0    0
0.0140528    0.0490567
0    0
0.0076138    0.0084168
0    0
0    0
0.0235345    2.44537
0    0
0.201207    0.131835
0    0.012587
0    0
3.2332    2419.1
25.3591    1848.09
1.24157    0.185647
0    0
0.0707284    0.380199
0.000926048    0.0291497
0.556667    258.285
0.00765224    0.0423483
0    0
0.071368    0.172312
0.283154    0.859825
0    0
0.00321627    0.0208772
0.0733062    0.035614
0.0428443    0.137591
0    0
0    0
0.0126421    0.0494119
0.103812    0.0389233
0    0.00153992
0.0640372    0.140712
0    0
0.00267758    0.710792
0    0
5.87483    20.4657
0.0127048    0.0583802
0.00750976    0.00219457
0    0
0.592349    0.386329
0    0
0.00407045    0
0    0.0393989
1.45026    1.18887

Here is the code which I used in R

t.test(int.genes[,1],int.genes[,2], paired=T)

     p-value = 0.0633


wilcox.test(int.genes[,1],int.genes[,2], paired=T)

p-value < 2.2e-16

[Michael]

show us the data or it didn't happen

[Vikas Bansal]

I have pasted the FPKM values of 585 genes and R code. Please have a look.

[David]

The way you are calculating your t-test is incorrect. The t-test has to be perform for every gene and not across a group of gene in two conditions. The way you are doing it now does not give a meaningful p-value.

[Asaf]

I disagree, I think that it's a legitimate question to ask and with much solid statistical ground than t-test for each gene separately (which is not possible statistically). However, I think you should remove the 0 0 rows, i.e. remove missing data. An assumption he makes here is that values from both conditions come from the same distribution (mean and variance), this should be tested.

[Michael]

It's a legitimate question, but not the right method. A test cannot be applied like this. Read up on the test statistics if you are in doubt.

[Asaf]

I still don't understand why not, he has 2 conditions and paired objects, and wants to know if the values in A are greater than in B. Why paired t-test or Wilcoxon won't fit here?

[Vikas Bansal]

So if it is a legitimate question but not the right method, which method would you suggest?

[Asaf]

What did you compare? Did you compare the differential expression of these genes to another (background) set of genes or the FPKMs in one condition to the FPKMs in another condition? If you did the second option than the test was not performed right, you should have used a paired test (like paired Wilcoxon). If you did the first then the results are indeed disturbing.

[Vikas Bansal]

Thanks for the reply. I did the second option. Now I performed the paired test again. The p values are 0.18 for paired t-test and 1.6e-15 for paired Wilcoxon test (Wilcoxon signed-rank test). Do you prefer paired Wilcoxon over paired t-test in this case?

[Asaf]

Still weird, I would check how the t-test you used estimates the variance and plot the values (condition A - condition B.) of all the genes to see if they are normally distributed around the 0 (otherwise every random group of genes will have significant results)

[Vikas Bansal]

I used Q-Q plots and Shapiro-Wilk test to check the normality of the FPKM values. Indeed, they are not normally distributed. So, I think, I should use Wilcoxon signed-rank test?

[Asaf]

non-parametric statistics makes less assumptions so it should theoretically always be preferred but it's weird that t-test gave such a different result. I would take a random set of genes and make these tests on them as well to see if the Wilcoxon can be trusted.

Answer 1

[David]

In order to test if one or more genes are significantly differentially expressed (SDE) between two conditions one can perform a t-test. The t-test will test the null hypothesis that there is no difference between the mean of the two populations. Usually, if the p-value is below the significance threshold chosen (say 0.05) you reject the null hypothesis and conclude that there is a significant difference between the means.

I am describing the t-test here to explain what you are doing. In the description you gave you are taking all the values for the 585 genes across two conditions and compare the two populations. The problem here is that you consider values from different feature (genes) as being repeated measure on the same experiment. This is like measuring a bunch of unrelated feature on day one and the same features on day two. For example, you measure the weather temperature, humidity, UV index etc.. for 2 days in a row and asking does the mean of these feature is different for day 1 and day 2... You can do the test but the conclusions are not very informative.

I think the confusion in all the comments come from the fact that this approach is not completely wrong but unconventional, questionable and results are not robust.

• Unconventional because gene expression experiments are designed to gain knowledge about individual genes that are SDE between two conditions. For example, you have 3 values for gene A coming from biological replicate for condition A and the 3 values for condition B. In this case you perform a t-test between condition A and B. Perform the same test for all genes tested and correct for multiple hypothesis testing.

• Questionable because using your approach you can cherry pick a list of gene and obtain either a significant difference or not. Also saying that a group of genes (i.e. your 585) has a significant difference in mean is not really informative. There could be one or two genes driving the difference observed. Additionally, the chance to obtain a significant difference between two conditions will be lost when the gene list increase. This is due to the normalization of the gene expression across experiment to make them comparable. This will certainly be true if you use a median polish procedure like in RMA-like methods.

• Not robust because the experiment you describe has N=1. You need replicates.

I hope this help.

[Vikas Bansal]

Thanks a ton for your answer. The reason I was doing the mean difference test (not significantly differentially expressed) because I have seen somewhere in the publications that researchers compare the FPKM (or intensities) values of same set of genes in 2 conditions and make a boxplot and put the P-value over these 2 boxplots (saying that mean expression is significantly higher in condition x).

Answer 2

[David]

Asaf is right the difference you are seeing are pretty extreme. Non-parametric test are more robust from departure from normality however here I would look at the data.

For example what is your number of replicate in each group?

Are your samples paired? You should only use paired t-test when your samples are. Usually biological replicate are not considered paired but patient data can sometime if it make sense. It has an influence on the conclusion implication.

You mention that you look only at a subset of the genes in the experiment but this does not mean you should not do multiple hypothesis testing correction.

Sharing your code would also help us help you.

Hope this help.

Answer 3

[Michael]

Well, for what are seemingly trying to do you used the wrong test.

[Vikas Bansal]

Could you please suggest which test should I use and why?

[Michael]

Sorry if I wasn't clear. What I mean is, a test for difference in means is not applicable across different genes. If you have replicates, you can test for differences between two conditions for the same gene but never across genes. This is because t test and Wilcox assume that all values come from the same distribution. This is arguably not the case for different genes, and therefore both results are bogus. I will explain more when I have a keyboard again.

Answer 4

[earonesty]

You need biological replicates to do differential expression. The number of replicates you need depends on the expected degree and stability of variation induced by the experiment relative to the control.

With "relatively homogenous" cell cultures that have low variability, 3 replicates (dishes, plates, etc) is often acceptable. You can't do the experiment once, dry down, and cut up the pellet into 3 pieces. You have to actually do the experiment 3 times.

For in-vivo experiments, you'll need lots more, and this also is dependent on the tissue type. Blood is relatively homogenous. But other hard tissues are dependent on the skill and repeatability of the researcher doing the sampling. Muscle sampling can contain blood, fat and other tissue that will affect expression. Also, we've found that finely blending tissue first helps even out data. So there are a lot of considerations. Again, a rule of thumb I have is 5 samples for blood, 7 or more for other tissues.

Don't forget that proper randomization during the experiment can mitigate false positives, but won't help you with sensitiviry. For that you need repeatable methods and increased numbers of samples.

One of the terrible things that has come out of RNA-seq is the rash of papers comparing 1 sample to another. That's because of programs like edgeR and deSeq which claim to be effective in this manner. They are not. You only have to think about the issues above (muscle mixed with more fat in one sample and less in another) to realize how silly this assumption is.