Counting number of duplicated reads from fastq file

Question

[mike]

Hello,

Which tool or method is available for counting or extracting number of duplicated reads from fastq files with paired reads? I have checked various tools which can only removes duplicated reads

Thanks

Answer 1

[Brian Bushnell]

Hi Mike,

If you want to deduplicate raw paired fastq files, I recommend trying dedupe.sh from the BBMap package. You can run it like this:

dedupe.sh in1=read1.fq in2=read2.fq out1=x1.fq out2=x2.fq ac=f

That will also print the exact number of duplicates removed.

Answer 2

[iraun]

You can remove duplicates (using picard, samtools or whatever) and then count how many reads are missing from the de-dupped file, no?

[iraun]

If you don't want to remove duplicates you can also, use samtools flags:

samtools view -c -f 1024 file.bam

[mike]

These tools are for aligned bam files.

[iraun]

Oh, sorry, my mistake. So, I would suggest you to map first. In my opinion, it is much better to map first and remove duplicates then. But, if you want to remove duplicates first, you could try fastx_collapse, remove the duplicates and count how many of them have you lost.

[Devon Ryan]

It should be noted that fastx_collapse only works on single-end reads (this is pretty common for tools like this).

[mike]

Okay thanks I will give it a shot.

[mike]

Fastx toolkit is not designed for paired end reads if I am not wrong

[Devon Ryan]

Unless you want to use these for assembly, it's generally fast enough to just align and remove/mark duplicates from the resulting BAM file.

Answer 3

[kaixian110]

HI , have you found any solutions to extract duplication reads from paired fastq files ?

[Brian Bushnell]

BBMap's dedupe program has an "outd" flag that will capture duplicate reads:

dedupe.sh in1=read1.fq in2=read2.fq out1=x1.fq out2=x2.fq ac=f outd=dupes.fq

Alternatively, you can use Clumpify:

clumpify.sh in=reads.fq out=clumped.fq markduplicates allduplicates

This command assumes paired reads are interleaved in a single file, although the upcoming release supports paired reads in twin files. The "allduplicates" flag will mark all copies as duplicates; if you remove that, all but one copy will be marked as duplicates (which is probably better for most purposes). The "optical" flag will mark only optical duplicates (rather than, say, PCR duplicates). Anyway, "clumped.fq" will contain all of the reads, but the duplicates will be marked with " duplicate". So you can then separate them like this:

filterbyname.sh in=clumped.fq out=dupes.fq include=t names=duplicate substring
filterbyname.sh in=clumped.fq out=unique.fq include=f names=duplicate substring

[GenoMax]

One can easily get interleaved data files for clumpify.sh by using another tool from BBMap: reformat.sh in1=R1.fg.gz in2=R2.fq.gz out=int.fq.gz.