Remove duplicate protein sequences having different fasta identifiers

Question

[utkarsh.sood]

Hello

I have 9058 faa files. some are having duplicate protein sequences but with unique different fasta identifiers. How these duplicate sequences can be removed?

Thanks!

[DG]

The answer below is fantastic. I'd point out that strictly speaking these aren't necessarily duplicate sequences. They could be if someone just made a mistake with FASTA headers, but chances are they are protein sequences of the same protein from different organisms, or strains, or isolates, etc. They may be identical sequences, and you may only want a single representative in those cases, in which case Sej's answer below with clustering will solve it. But for clarity, Identical sequences doesn't necessarily equate to duplicate sequence.

Answer 1

[Sej Modha]

You can use cd-hit with 1.0 identity and redundancy 1 (-t parameter) to do this.

[ALchEmiXt]

yep cluster the sequences. One representative will be kept.