Removing redundant sequences

Question

[kutubjoy]

I have a fasta file containing many protein sequences with identifier. How can I remove the redundant sequences using HMMscan

[Ram]

Why use HMMscan to remove redundant sequences? There are a gazillion deduplication tools out there - just search the forum for FASTA deduplication.

[Kurban]

Hello guys

I have more than 70,000 protein sequences come from 65 animal species, most of them are TFs. So some of them might be homologous. I like to use CD-HIT to remove the them. But which similarity threshold should I use?

Any suggestion?

[Eslam Samir]

Here is my free program on Github Sequence database curator (https://github.com/Eslam-Samir-Ragab/Sequence-database-curator)

It is a very fast program and it can deal with:

1. Nucleotide sequences
2. Protein sequences

It can work under Operating systems:

1. Windows
2. Mac
3. Linux

It also works for:

1. Fasta format
2. Fastq format

Best Regards

[Ram]

I see that you've created a Tool type post for your tool. Please do not spam threads with ads for your tool.

Answer 1

[nterhoeven]

I wrote a perl script to do that. It is called remove_duplicates and you can find it here:

https://github.com/nterhoeven/sequence_processing

Answer 2

[h.mon]

As Ram suggested, there are several tools to do this, one is cd-hit.

[Ram]

This is an old question :)

[h.mon]

Not for my standards, I even replied to 3-4 year old questions ;-)