Tools to remove duplicate or substring reads

Question

[chjiao3456]

Is there any efficient tool to remove substring reads or duplicate reads from NGS data set? I know that readjoiner could remove the duplicated reads, but seems not work on substring reads. Thanks.

Example: Duplicates: read1: AGTCAT read2: AGTCAT In this case, only one read will be kept.

Substring: read1: GTCA read2: AGTCAT In this case, read1 will be removed.

Answer 1

[Brian Bushnell]

The most efficient tool for this purpose is Dedupe from the BBMap package. However, it requires all reads to be stored in memory, so it needs a lot of memory. Can you explain in more detail what you are trying to do?

[chjiao3456]

Thanks for your help. I have added examples in the question.

Answer 2

[chjiao3456]

Just noticed that SGA tool is able to do this. Collapse Reads That Are Substrings Of Other Reads In Same Library