I am testing several RADseq SNP datasets for Fst outliers using BayeScan 2.1. These datasets represent the same sampling of individuals, but with different numbers of SNPs included and either 2 or 3 populations being tested.
I convert .genepop files to the .bayescan file format using PGDSpider. I cannot see any differences among the genepop or bayescan files.
However, when I run BayeScan (using server commandline or GUI on PC), five out of six of my datasets result in Fst values of ~0.5 rather than ~0.01. The latter range seems far more likely, given Fst estimates made in other software such as STACKS and adegenet in R, and because my populations are mostly homogenous, and from a marine organism.
Does anyone have an explanation for why BayeScan is estimating such high Fst values for all but one of my datasets?
I should also add that BayeScan is also failing to calculate alpha values in the .sel files for the affected datasets.