fastq file error

Question

[sc@791]

Hi! Members,

I am using a perl program for Transposable Element analysis, which required unmapped fastq reads from Bismark output. My original raw data was paired end and I used the following command line for Bismark mapping:

bismark_v0.21.0/bismark ~/Bismark/Genome -bowtie2 --ambiguous --non_directional -unmapped -R 10 score_min L,0,0.6 -N 1 -1 Epiril_22C_R1_L1/Epiril368_22_C_R1_L001_R1_val_1.fq.gz -2 Epiril_22C_R1_L1/Epiril368_22_C_R1_L001_R2_val_1.fq.gz -o New_Output/Epiril368_22C_Rep1_L1.bam

The output from this command gave me unmapped read1 and read2 files which I concatenate as:

cat unmapped.read1.fq unmapped.read2.fq > unmapped.fq

Now when I am using perl program epiTEome, it giving me error:

perl epiTEome.pl -gff Analysis/tair10TEs.gff3 -ref Analysis/TAIR10_chr_all.epiTEome.masked.fasta -un trial/Epiril368_4_C_R1_L001_R1_val_1.fq.gz_unmapped_reads_1.fq -t Analysis/teid.lst

Possible precedence issue with control flow operator at epiTEome.pl line 482.
INFO epiTEome.pl Fri Jun 28 09:49:46 2019 Start program!
INFO epiTEome.pl Fri Jun 28 09:49:46 2019 Run Module: readGffFile!

INFO epiTEome.pl Fri Jun 28 09:49:49 2019 STEP 1: read ends mapping.
INFO epiTEome.pl Fri Jun 28 09:49:49 2019 Run Module: splitFastq!

------------- EXCEPTION: Bio::Root::Exception -------------
MSG: Missing sequence and/or quality data; line: 4
STACK: Error::throw
STACK: Bio::Root::Root::throw /usr/local/share/perl/5.26.1/Bio/Root/Root.pm:449
STACK: Bio::SeqIO::fastq::next_dataset /usr/local/share/perl/5.26.1/Bio/SeqIO/fastq.pm:121
STACK: main::splitFastq epiTEome.pl:501
STACK: main::main epiTEome.pl:93
STACK: epiTEome.pl:55

What I understood from the error is I have problem in unmapped.fq file.

Any suggestion to quality check my unmapped fastq files.

Thank you! Regards

[GenoMax]

Run validateFiles utility from Jim Kent's UCSC tools (after download add execute permissions, chmod a+x validateFiles) to make sure your fastq files are in proper format.

Are you sure that the perl program you are using expects a combined file (like the one you made R1+R2 at end of R1 file)?

[sc@791]

Thank you for prompt suggestions.

here is the report from validateFiles:

./validateFiles -type=fastq ~/epiTEome/Analysis/Unmapped_reads/Epiril368_4_C_R1_L001_unmapped.fastq** 
Error [file=/home/sc/epiTEome/Analysis/Unmapped_reads/Epiril368_4_C_R1_L001_unmapped.fastq, line=2]: invalid DNA chars in sequence(+)
Aborting .. found 1 error

Yes the program can take concatenate fastq files, however, the error is occurring even I used file without merging.

the head of fastq file is:

~/epiTEome/Analysis/Unmapped_reads/Epiril368_4_C_R1_L001_unmapped.fastq 
@K00230:34:HMJJFBBXX:1:1101:2869:1226_1:N:0:CAAAANNTAGATAAGGAAAAAAAAAAAATTATTTTAAGTTGTTATNATTTATTAGATATTAATTAATAAANATTATGNANATAAGATATNAGTTNGTTAATAGAGAATAAGAATATTATAAATTTAATGATTAATTAATAANATGAAAAAAGGGA
+
#AA-FJJJFFJFJJJJJJJJFJJJJJAJJJFAF-AFFJJ#JAJJJJ<FAF<JFJJJJFJJFAJJ#JJJFAJ#F#J<F-<F-J-#<--<#<<FJJAF<FFJF-JJ<JJ<JFFJ-AJJ----<--<-<FJA-AFAJJ#FA-7AA<<F7--A
@K00230:34:HMJJFBBXX:1:1101:3031:1226_1:N:0:CAAAANNAATGGTTCGTATCTGAGTTTTTTTATCTGAATAATTTTNTATTTTTTTTTTGAAGAAAAAAATNATTAAANTNTTGATATTTNGAGANATTTTTAATTCGAATTTTTTTTTTTGTAACCTTTTAGTTTTTCGATNTTTTTTTTTTCTTT
+
#AAAFJJJFJJJJJJJJJJJFFJJJJJJ<JJFJJJJJJJ#JJJJFJJJJJJJFFFJJJJJJFJF#JJJJFJ#J#JJJFJAJJJ#-<AA#7FJJJJJFJF-AJ7FJJJJJJJJJJJF7A-7-7F-A-FJ<JJ-7FF#FJJJJJF<FJ-7-F
@K00230:34:HMJJFBBXX:1:1101:3417:1226_1:N:0:CAAAANNGAAGTTTGGTTATTGTTTTGGAGTCGAATATGATTTGANGTTATGTGTATGATTGAGTATAAGNATTTAANTNGCAACCGGANTTTANAAGTTTAAGTAGTGTGTTTTTGTTAGAATATACAAAGTTAAAGATTNATATGGATTTTGGT
+
#AAFFJJJJJJAJJJJJJJJJJJJJJJJJJJJJJJJJJJ#JJJJJJFJFJJJJJJJJJJJJJJJ#JJJJJJ#J#JJFJJJJJF#JJJJ#JJJ<JJJJJFJJJJAJFJAJJJJFJJJJJJJJJJJJFJJJJJJJJJ#JJJJJJF7FFJJJJ
@K00230:34:HMJJFBBXX:1:1101:3945:1226_1:N:0:CAAAANNATATTTTGTTAAAGATTTTTTTTTGTTTTTGTTTTTTTNTTAAAGAATATAATTTTAATAAATNTGGATANANATAGTGAATNTGTTNTGGTTGAGTTAAAATTAGAATTATATTGTAAATTTTGTGTATTTAGNTAAATTTTAAAAT

Thank you! Regards

[GenoMax]

As you can see somehow your sequence (line 2) has gotten appended at the end of line 1 (fastq header).

How did that happen?

A good fastq record should look like this.

@K00230:34:HMJJFBBXX:1:1101:2869:1226_1:N:0:CAAAAN
NTAGATAAGGAAAAAAAAAAAATTATTTTAAGTTGTTATNATTTATTAGATATTAATTAATAAANATTATGNANATAAGATATNAGTTNGTTAATAGAGAATAAGAATATTATAAATTTAATGATTAATTAATAANATGAAAAAAGGGA
+
#AA-FJJJFFJFJJJJJJJJFJJJJJAJJJFAF-AFFJJ#JAJJJJ<FAF<JFJJJJFJJFAJJ#JJJFAJ#F#J<F-<F-J-#<--<#<<FJJAF<FFJF-JJ<JJ<JFFJ-AJJ----<--<-<FJA-AFAJJ#FA-7AA<<F7--A

That is a strange choice of index (if CAAAAN is real).

---

Please use the formatting bar (especially the code option) to present your post better. I've done it for you this time.

Thank you!

[sc@791]

Many thanks! for your suggestions and corrections. I will use make sure to use formatting bar in future.

I don't have any idea how did it happen. I think I made some mistake while mapping the reads.

If you any suggestions to correct it please let me know. Otherwise I have to start from Indexing genome and mapping.

Thank you! Regards

[GenoMax]

I am not sure how to tell you. If your original files were fine then just running bismark should not have done this. You would need to backtrack and re-do things as needed.

Answer 1

[colin.kern]

Have you looked at your fastq file? Do you know what the fastq format should look like? It might help if you can update your question with the output of "head unmapped.fq". Every read in a fastq file is represented by 4 lines:

@J00113:218:HGGJVBBXX:5:1101:27610:1226 1:N:0:NTTGTA
ACCTATGAAAACTCTNAAAGAAATGTAAGTTTTTATTANATTCNNGTCAGTTACTGANTTTAATCTTCATCAGNTACNTGTGGTAATTTA
+
AAA-F7FJJFJJJJF#FJJJ<JJJJJJJFJFJJJJJJJ#JJJJ##JJJJJJJFJJJJ#JJFFJJJFJFJJJJJ#JJ<#JJJJJJJJJJJJ

The first line in a unique identifier for the read, with some metadata information. The second line is the nucleotide sequence. The third line is always "+", and the fourth line is the quality information. The important part of the error you are getting is "Missing sequence and/or quality data; line: 4", so it would help to see what the first couple reads of the file look like.

[sc@791]

my header looks like:

head ~/epiTEome/Analysis/Unmapped_reads/Epiril368_4_C_R1_L001_unmapped.fastq 
@K00230:34:HMJJFBBXX:1:1101:2869:1226_1:N:0:CAAAANNTAGATAAGGAAAAAAAAAAAATTATTTTAAGTTGTTATNATTTATTAGATATTAATTAATAAANATTATGNANATAAGATATNAGTTNGTTAATAGAGAATAAGAATATTATAAATTTAATGATTAATTAATAANATGAAAAAAGGGA
+
#AA-FJJJFFJFJJJJJJJJFJJJJJAJJJFAF-AFFJJ#JAJJJJ<FAF<JFJJJJFJJFAJJ#JJJFAJ#F#J<F-<F-J-#<--<#<<FJJAF<FFJF-JJ<JJ<JFFJ-AJJ----<--<-<FJA-AFAJJ#FA-7AA<<F7--A
@K00230:34:HMJJFBBXX:1:1101:3031:1226_1:N:0:CAAAANNAATGGTTCGTATCTGAGTTTTTTTATCTGAATAATTTTNTATTTTTTTTTTGAAGAAAAAAATNATTAAANTNTTGATATTTNGAGANATTTTTAATTCGAATTTTTTTTTTTGTAACCTTTTAGTTTTTCGATNTTTTTTTTTTCTTT
+
#AAAFJJJFJJJJJJJJJJJFFJJJJJJ<JJFJJJJJJJ#JJJJFJJJJJJJFFFJJJJJJFJF#JJJJFJ#J#JJJFJAJJJ#-<AA#7FJJJJJFJF-AJ7FJJJJJJJJJJJF7A-7-7F-A-FJ<JJ-7FF#FJJJJJF<FJ-7-F
@K00230:34:HMJJFBBXX:1:1101:3417:1226_1:N:0:CAAAANNGAAGTTTGGTTATTGTTTTGGAGTCGAATATGATTTGANGTTATGTGTATGATTGAGTATAAGNATTTAANTNGCAACCGGANTTTANAAGTTTAAGTAGTGTGTTTTTGTTAGAATATACAAAGTTAAAGATTNATATGGATTTTGGT
+
#AAFFJJJJJJAJJJJJJJJJJJJJJJJJJJJJJJJJJJ#JJJJJJFJFJJJJJJJJJJJJJJJ#JJJJJJ#J#JJFJJJJJF#JJJJ#JJJ<JJJJJFJJJJAJFJAJJJJFJJJJJJJJJJJJFJJJJJJJJJ#JJJJJJF7FFJJJJ
@K00230:34:HMJJFBBXX:1:1101:3945:1226_1:N:0:CAAAANNATATTTTGTTAAAGATTTTTTTTTGTTTTTGTTTTTTTNTTAAAGAATATAATTTTAATAAATNTGGATANANATAGTGAATNTGTTNTGGTTGAGTTAAAATTAGAATTATATTGTAAATTTTGTGTATTTAGNTAAATTTTAAAAT

[colin.kern]

Ok, the problem is that the ID and the sequence lines are being combined.

[sc@791]

Many thanks!

I don't have any idea how did it happen. I think I made some mistake while mapping the reads.

If you any suggestions to correct it please let me know. Otherwise I have to start from Indexing genome and mapping.

Thank you! Regards