Detecting outlier loci from pairwise Fst values

Question

[Earendil]

Having calculated pairwise Fst values with vcftools, I now need to find the threshold for outlier loci. I've decided to follow the second suggestion from an answer of this thread: Calculating statistically significant outlier for Pairwise Fst obtained from VCFTools which is:

2. Permute your genotypes and re-run Fst many times. This would be considered an empirical p-value, or probability.

Since I am having no statistical background, would there be a simple explanation on how to implement this?

[Shangzhe Zhang]

Dear Earendil,

I'd like to know that if you solve the problem. I'm new to NGS analysis and I'm stuck in this problem. Hope for your help!

[Earendil]

Dear Shangzhe,

That was long time ago, I didn't find out how to do that so instead I used the software Bayescan which directly detects Fst outliers, you would might want to take a look at that.

[Shangzhe Zhang]

Dear Earendil,

Thanks for you reply. Coincidentally one of my colleague used this method to find the outliers of Fst.

On my opinion this method is useful when your windowed-Fst data normally distributed and its mean is almost 1, which is difficult to identify the outliers using the p-value against the normal distribution defined by its mean and standard distribution.

The method is actually called multiple testing. For our data, this method avoid the bias due to the windows with few SNP pairs but with the Fst value relatively high.

Maybe this is ambiguous, I copy the link of the article of my colleague and the article of the method below. Hope these can help:

• http://www.pnas.org/content/115/2/E236 (see the "method" part)
• https://www.researchgate.net/publication/40688145_How_does_multiple_testing_correction_work_Nat_Biotechnol

And thank you again. I will see Bayescan.