Removing Redundant Amino Acid Sequences From Fasta - Tools And Algorithms

Question

[Jacek]

Is there any tool that can be used to remove redundant amino acid sequences from Fasta? I know fastx_collapser from fastx-toolkit (How to remove the same sequences in the FASTA files?), but it works on nucleotide sequences only. I am looking for a similar tool.

Are there any widely known algorithms for that purpose? Is using Levenshtein distance, pair distance or similar a good idea?

[Jacek]

All the answers are great, thank you guys. However, I would like to know more about algorithms that can be used. Share your ideas.

Answer 1

[Pawel Szczesny]

If using prove-that-you're-academic-ware doesn't bother you, you should definitely try UCLUST. Its author claims UCLUST is definitely superior to CD-HIT, and while I would take it with a grain of salt, much smaller requirements at much higher speed are hard to ignore.

Answer 2

[Neilfws]

Many of the answers to the question that you mentioned will work fine for amino acid sequence.

You should definitely look at CD-HIT. Although not updated for some time it works well and is still widely-used; for example, to create NR databases at UniProt and the PDB.

[Pawel Szczesny]

It is constantly developed (http://code.google.com/p/cdhit/updates/list ) but indeed the last stable release was quite some time ago.

Answer 3

[Ijessie]

Days ago, I wrote a Python script to remove redundant nucleotide sequences from FASTA. Here is the script:

import re

pattern_newline = re.compile(r'([A-Z-]{80})(?=[A-Z-])')

seqs = {}

with open('example.fasta') as fin:
    data = fin.read()
    for m in re.finditer(r'>(\w+)[^\n]*([^>]*)', data):
        seq = ''.join(m.group(2).split())
        id = m.group(1)
        if seq not in seqs: seqs[seq] = []
        seqs[seq].append(id)

with open('unique.fasta', 'w') as fout:
for seq, ids in seqs.iteritems():
    fout.write('>%s\n%s\n' % (ids[0], pattern_newline.sub(r'\1\n', seq)))

It works on amino acid sequences, too. Wish it helps.

Answer 4

[Eslam Samir]

Here is my free program on Github Sequence database curator (https://github.com/Eslam-Samir-Ragab/Sequence-database-curator)

It is a very fast program and it can deal with:

1. Nucleotide sequences
2. Protein sequences

It can work under Operating systems:

1. Windows
2. Mac
3. Linux

It also works for:

1. Fasta format
2. Fastq format

Best Regards