Removing PCR duplicates from .fastq without .bam alignment

Question

[Adrian Pelin]

Hello,

I have an old dataset from 2010 of PE illumina 54bp reads with a lot of PCR duplicates. These pairs of duplicates are very obvious, they are exactly the same read sequence forward and reverse present several times with different read names.

I know how to get rid of them using a bam alignment/mapping, but I am interested in methods to remove these without an alignment since I am interested on doing analysis on all reads, not just those that align to the genome.

What are some available approaches that take as input fastq and output fastq?

Thank you,

Adrian

[komal.rathi]

Also, PRINSEQ

[Adrian Pelin]

This worked:

perl prinseq-lite.pl -fastq ~/Encephalitozoon/Eromalae/100611_s_4_1_seq_GDR-7.fastq -fastq2 ~/Encephalitozoon/Eromalae/100611_s_4_2_seq_GDR-7.fastq -phred64 -derep 1

[lkmklsmn]

Check out FastUniq

[Adrian Pelin]

perl prinseq-lite.pl -fastq ~/Encephalitozoon/Eromalae/100611_s_4_1_seq_GDR-7.fastq -fastq2 ~/Encephalitozoon/Eromalae/100611_s_4_2_seq_GDR-7.fastq -phred64 -derep 1

That's a bit odd that the max is 1000 pairs.

[Charles Plessy]

Just for the record: FastUniq can not account for sequencing errors (which can be a strong limitation). Here is a quote from the authors' article (Xu _et al._, 2012).

There were some differences in levels of duplicates identified by FastUniq and Picard Markduplicates that were caused by the different criteria in read pair comparisons (Figure 3A, Table 1). Of them, FastUniq compares read pairs on the basis of sequences only, and it is sensitive to SNPs caused by heterozygous or sequencing errors.

[kaixian110]

Hi , do you know the same function tools written by python ?

Answer 1

[Brian Bushnell]

Clumpify can mark or remove duplicate reads very efficiently without alignment:

clumpify.sh in=reads.fq out=deduped.fq dedupe