Remove duplicates in two databases

Question

[boudica5]

Hello. I have two databases (.fas) that I have to unify in only one. Some sequences are present in both DB, I want to keep one copy and delete the other. Instead of removing manually one by one, is there any way to select all duplicates and remove them? Thank you very much.

[Pierre Lindenbaum]

'.fas' = fasta file ? (= just a file, not a database)

[boudica5]

Yes, I have two files

[Pierre Lindenbaum]

so this question has been asked a gazillon times on biostars: How To Remove The Same Sequences In The Fasta Files? , Remove Redundant Sequences Fasta , How To Remove The Identical Sequences In The Fasta Files? , Useful Bash Commands To Handle Fasta Files , Remove Redundant Sequences Fasta , ....

Answer 1

[GenoMax]

Dedupe.sh from BBMap should be able to do this. It will accept fasta files. Input is expected to be DNA.

[boudica5]

Thank you, do you know if it accepts fasta files with protein sequences?

Also, citing Brian Bushnell from the link you gave me "However, I do have another program, filterbyname.sh, that can remove all sequences from a file that either share or don't share names with sequences in another file", again do you know if it accepts protein sequences?

My best regards

[Brian Bushnell]

Dedupe strictly works on DNA (or RNA) sequences. However, filterbyname.sh should work on protein sequences if you use the aminoin flag... I've just never tested it. Note that, of course, it will only address the situations where sequence share identical or similar names (depending on the mode); it ignores the sequence itself.

[boudica5]

I´ll try it! Thank you all for your kind responses

[GenoMax]

In that case you will need to use one of the solutions indicated by Pierre (though most of those appear to be for DNA too). BBMap tools are for DNA sequences AFAIK. I have edited my original post.