outliers vs false positives LOSITAN

Question

[lccuervoa]

Hello!

I wonder if you may clarify me something.

I am running LOSITAN and when I get the results table, I observe that some loci are highlighted in red and yellow, and among those, only few say "outlier"... why does it happen? Are those highlighted but without the word "outlier" most likely false positives?

Thanks in advance!

Laura

Answer 1

[tiagoantao]

The outlier color and naming comes from the original Beaumont and Nichols paper. So after the FST/He surface is computed using your average FST, total number of populations and mutation model, each loci is evaluated against the 2D surface. Think of it as an empirical Confidence Interval in two dimensions.

A completely different approach is the FDR one (they can, and do give, different results). Here we use the Benjamini Hochberg procedure. I think my Mcheza paper has the reference for this. The colors and "outlier" label DO NOT use the FDR, but the old approach.

Which to choose is up to you and your problem (some people would say the FDR approach makes more sense, but I will stay neutral here). In any case do not forget that the candidate selected loci is just that. This is a statistical suggestion to a biological problem. Whichever list you get (from LOSITAN or any other program), take it with a grain of salt.