my fastq file have two same reads' name

Question

[jiangzhiyong12]

Hi, i have PE resequencing data, but in some fastq file, it has two same reads' names, now i want to delete one of the two, so i want some suggestions from you all. Thank you.

[GenoMax]

That is not possible unless the files were messed with in some way/mistreated. How did you determine (which program/error message) that you have this condition?

[jiangzhiyong12]

when i run the GATK workflow when markduplicates, i got this error:

Exception in thread "main" htsjdk.samtools.SAMException: Value was put into PairInfoMap more than once. 1: HWI-ST1307:159:C48TVACXX:7:1109:1787:63474
at htsjdk.samtools.CoordinateSortedPairInfoMap.ensureSequenceLoaded(CoordinateSortedPairInfoMap.java:133)
at htsjdk.samtools.CoordinateSortedPairInfoMap.remove(CoordinateSortedPairInfoMap.java:86)
at htsjdk.samtools.SamFileValidator$CoordinateSortedPairEndInfoMap.remove(SamFileValidator.java:765)
at htsjdk.samtools.SamFileValidator.validateMateFields(SamFileValidator.java:499)
at htsjdk.samtools.SamFileValidator.validateSamRecordsAndQualityFormat(SamFileValidator.java:297)
at htsjdk.samtools.SamFileValidator.validateSamFile(SamFileValidator.java:215)
at htsjdk.samtools.SamFileValidator.validateSamFileSummary(SamFileValidator.java:143)
at picard.sam.ValidateSamFile.doWork(ValidateSamFile.java:196)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:205)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:94)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:104)

so i search the read name HWI-ST1307:159:C48TVACXX:7:1109:1787:63474 in my fastq file ,i got two same reads' names, i also search in my .sam file, i got 4 same reads' names, weired......which i thought it's the fault of the sequencing company, maybe they just copy any data within the same file, and put them together.......

[Dan D]

Can you use grep -A and tell us if the content of the two reads with identical names is the same in terms of sequence and quality scores?

[jiangzhiyong12]

i search my sam file

$more 11.sam | grep HWI-ST1307:159:C48TVACXX:7:1116:16586:11978

Result:

HWI-ST1307:159:C48TVACXX:7:1116:16586:11978 65  gi|539359185|ref|NW_005087554.1|    565080  1   52M gi|539359184|ref|NW_005087555.1|    40387192    0   AGGAAGGAAGGAAGGAAGGAAGGAAGGAAGGAAGGAAGGAAGGAAGGAAGGA    BCCFFFFFHHHDDIJGBGGHDGGGGHGIGGGFDEGG;DHA?FEB=F=@@FGE    AS:i:0  XS:i:0  XN:i:0  XM:i:0  XO:i:0  XG:i:0  NM:i:0  MD:Z:52 YT:Z:UP
HWI-ST1307:159:C48TVACXX:7:1116:16586:11978 129 gi|539359184|ref|NW_005087555.1|    40387192    0   10M4I12M4I31M4D23M  gi|539359185|ref|NW_005087554.1|    565080  0   CCTTCCGCCACTTCCTTCCTTCCGCCACTTACTTCCTTCCTCCACTTCCTTCCTTCCGCCACTTCCTTCCGCCACTTCCTTCCG    @@@FFFFFHGHHHJJIJJJJJJIIIJAHHI>DHIJJIIJGIGGHEHGIGGBFAGGHIBHIA:CHECDE?;9>ABBACD;@CC?B    AS:i:-51    XS:i:-51XN:i:0  XM:i:0  XO:i:3  XG:i:12 NM:i:12 MD:Z:53^CTTC23  YT:Z:UP
HWI-ST1307:159:C48TVACXX:7:1116:16586:11978 65  gi|539359185|ref|NW_005087554.1|    565080  1   52M gi|539359184|ref|NW_005087555.1|    40387192    0   AGGAAGGAAGGAAGGAAGGAAGGAAGGAAGGAAGGAAGGAAGGAAGGAAGGA    BCCFFFFFHHHDDIJGBGGHDGGGGHGIGGGFDEGG;DHA?FEB=F=@@FGE    AS:i:0  XS:i:0  XN:i:0  XM:i:0  XO:i:0  XG:i:0  NM:i:0  MD:Z:52 YT:Z:UP
HWI-ST1307:159:C48TVACXX:7:1116:16586:11978 129 gi|539359184|ref|NW_005087555.1|    40387192    0   10M4I12M4I31M4D23M  gi|539359185|ref|NW_005087554.1|    565080  0   CCTTCCGCCACTTCCTTCCTTCCGCCACTTACTTCCTTCCTCCACTTCCTTCCTTCCGCCACTTCCTTCCGCCACTTCCTTCCG    @@@FFFFFHGHHHJJIJJJJJJIIIJAHHI>DHIJJIIJGIGGHEHGIGGBFAGGHIBHIA:CHECDE?;9>ABBACD;@CC?B    AS:i:-51    XS:i:-51XN:i:0  XM:i:0  XO:i:3  XG:i:12 NM:i:12 MD:Z:53^CTTC23  YT:Z:UP

I really don'y know the reason. any help would be appreciated

[GenoMax]

That is odd. If you have not done anything to your SAM file then it is likely that your original fastq file has that read in there two times. Can you check that next?

[jiangzhiyong12]

It' true, I do have two same reads' name in my original fastq file. I get my fastq file reads' name and to get the unique reads' name, more weired thing is, 43063238(total reads' name) - 24218735(unique reads' name) = 18844503(duplicates' reads' name).......I don't understand......

[GenoMax]

So the problem is much bigger than you expected. If the sequence is identical for the duplicate reads then you will have to deduplicate them or get a new copy of the original data.

[jiangzhiyong12]

Yes, you are right, i deduplicate them, just get unique reads, with the next command: $seqtk subseq /disk5/jiangzy/bowtie2/trimmomatic/1_1_clean.fastq remaining_1.list > 1_1.remain.fastq just got 2.71G fastq file, compare to the original data 10.22G. then i used the FastQC tools to get the info of my data. here is the most weired thing: https://ibb.co/fDd3e6 https://ibb.co/jvrOe6 https://ibb.co/nvdQsR https://ibb.co/bWGdCR https://ibb.co/hCKbz6 https://ibb.co/jYWpK6

Still thank you.

[rohitsatyam102]

Hi @jiangzhiyong12

Were you able to sort this out? I am having a similar kind of issue ...