How to find identical sequences in genome fasta file (by Python or any possible program) ?

Question

[Sony]

Hello everyone,

I have a genome fasta file which has 16,941 sequences. Here are example of my "genome.fasta":

>scf7180000026027
GAATGCATACTGCATCGATA

>scf7180000026028
CATAAAACGTCTCCATCGCT

>scf7180000026029
TGCCCAAGTTGTGAAGTGTC

>scf7180000026030
TGCCCAAGTTGTGAAGTGTC

I want to find identical sequences in this genome fasta file, and return their ids. My final purpose are find and remove any identical sequences present in my genome fasta file.

Thank you everyone for any suggestion.

[Pierre Lindenbaum]

• How To Remove The Same Sequences In The Fasta Files?
• Remove duplicate in fasta file based on sequence
• How to make sure there is no duplicate sequence in a fasta file?

Answer 1

[GenoMax]

My final purpose are find and remove any identical sequences present in my genome fasta file.

You can use clumpify.sh from BBMap suite for this --> Introducing Clumpify: Create 30% Smaller, Faster Gzipped Fastq Files. And remove duplicates. It will accept fasta format sequences.

clumpify.sh -Xmx10g in=your_file.fa out=deduped_file.fa dedupe subs=0

subs=0 does perfect matches. Increase that number to allow mismatches.

You can use addcopies to mark headers with counts of sequences found like so

>scf7180000026027
GAATGCATACTGCATCGATA
>scf7180000026029 copies=3
TGCCCAAGTTGTGAAGTGTC
>scf7180000026028
CATAAAACGTCTCCATCGCT