how to detect and get genes with outlier expression from RNA-seq

Question

[rgescudero]

I am interested in searching for genes having outlier overexpression from RNA-seq samples (hundreds). I have checked for outlier analysis tools, but most are focused on detecting outlier samples instead of outlier genes. Also, I would like to known which expression values (FPKM, RPKM, TPM, CPM, etc) are better to detect outlier expression. I would thank any help on tools, recommendations, etc.

[h.mon]

I don't completely understand what you want to do:

Do you want genes showing consistent higher or lower expression compared to the other genes - that is, for a particular hypothetical gene, all samples have higher or lower expression compared to the other genes. edit: strictly speaking, these wouldn't be outliers.

Or do you want genes that, in one or a few samples, have a higher or lower expression compared to the same gene from other samples?

[rgescudero]

Second option: genes having higher outlier expression in a few samples compared with same genes in other samples. I'm interested in outlier overexpressed genes that may exist in some samples. It would be an unsupervised analysis because I do not have a priori information about which samples would have outlier expression of which genes.

Any help will be welcome

Answer 1

[h.mon]

DESeq2 uses Cook's distance to identify count outiers, check this vignette. The DESeq() function automatically replaces outlier counts and refits the GLM, however the vignette shows how you can identify outlier counts. Keep in mind you need a good number of samples per group to have reliable detection of outliers - which probably won't be a problem in your case.

[rgescudero]

Many thanks. I'll check it