how can I remove duplicated variants from vcf file?

Question

[kk.mahsa]

How can I remove duplicated variants from vcf file? I googled and searched in biostars history but I did not find any way to do it.

[Shicheng Guo]

Take 1000 Genome phase 3 data as the example:

bcftools norm -d both --threads=32 ALL.chr1.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz -O z  -o chr1.vcf.gz

[Shicheng Guo]

Terrible, still have duplicates: bcftools norm

Warning: Nonmissing nonmale Y chromosome genotype(s) present; many commandstreat these as missing.
Error: Duplicate ID '.'.

[smho]

Would you please explain more what do you mean by duplicated variants? Do you observe two lines in your VCF file that are exactly the same?

[kk.mahsa]

yes i mean is what you say and i want to keep one of the duplicate variants and remove the rest. in fact i intend to remove variants that are same in scoffold id and pos and keep one of them.

Answer 1

[Pierre Lindenbaum]

. in fact i intend to remove variants that are same in scoffold id and pos and keep one of them.

I strongly suggest you also use the REF information...

sort on CHROM/POS/REF. using awk create a KEY=CHROM\tPOS\REF, print the line if the key wasn't found previously

LC_ALL=C sort -t $'\t' -k1,1 -k2,2n -k4,4  input.vcf |\
awk -F '\t' '/^#/ {print;prev="";next;} {key=sprintf("%s\t%s\t%s",$1,$2,$4);if(key==prev) next;print;prev=key;}'

edit: added 'next; ' for VCF header.

[kk.mahsa]

thanks Pierre for your answer, i ran your cammand and get an vcf file as output but when used bcftools stats i got this error.

Failed to open output.vcf: unknown file type

why bcftools can not regognize output as a vcf file? i need to output file for downstream analysis as vcf file

[Pierre Lindenbaum]

ah yes, sorry it's because, sort messed-up the VCF header and ##fileformat= is not anymore the first line.

please try:

( grep  '^#' input.vcf ; grep -v "^#" input.vcf | LC_ALL=C sort -t $'\t' -k1,1 -k2,2n -k4,4 | awk -F '\t' 'BEGIN{ prev="";} {key=sprintf("%s\t%s\t%s",$1,$2,$4);if(key==prev) next;print;prev=key;}' )  > out.vcf

[kk.mahsa]

your answer was really helpfull, thank you so much Pierre. it worked

Answer 2

[cpad0112]

use vcfuniq or bcftools norm (with -d option) to remove duplicates

[Shicheng Guo]

bcftools norm left-align and normalize indels

Yes. It is left-align the alleles and then if the start coordinate is same then remove one of them, right?

[kk.mahsa]

thank you capd0112, i used bcftools norm and it worked.

[Pierre Lindenbaum]

bcftools normis new to me, thanks !

Answer 3

[Kevin Blighe]

More options (just adding to keep threads linked based on common information): A: Remove duplicate SNPs only based on SNP ID in bcftools

Kevin

Answer 4

[summerday1112]

Do you know how to remove duplicated variants from vcf file(ALL.chr1.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz), I am struggling for it.