Error: Help understand Trimmomatic ZLIB input stream error

Question

[James Reeve]

I've been running Trimmomatic of paried fastq.gz files and come across a weird error. It's particularly odd since the program finished running and the output looks normal. Can someone please give me a simplified explanation of what this error means? Furthermore, should I just ignore the message and keep the output?

Command:

 java -jar $TRIMMOMATIC PE -threads 12 -phred33 \
   -trimlog file/path/trimlog.txt \
   file/path/input_1.fastq.gz file/path/input_2.fastq.gz \
   -baseout file/path/output.fastq.gz \
   SLIDINGWINDOW:5:20 TRAILING:30 MINLEN:90

Error Message:

java.io.EOFException: Unexpected end of ZLIB input stream
    at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:240)
    at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
    at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117)
    at org.usadellab.trimmomatic.util.ConcatGZIPInputStream.read(ConcatGZIPInputStream.java:73)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.fill(BufferedReader.java:161)
    at java.io.BufferedReader.readLine(BufferedReader.java:324)
    at java.io.BufferedReader.readLine(BufferedReader.java:389)
    at org.usadellab.trimmomatic.fastq.FastqParser.parseOne(FastqParser.java:85)
    at org.usadellab.trimmomatic.fastq.FastqParser.next(FastqParser.java:179)
    at org.usadellab.trimmomatic.threading.ParserWorker.run(ParserWorker.java:42)
    at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-0" java.lang.RuntimeException: java.io.EOFException: Unexpected end of ZLIB input stream
    at org.usadellab.trimmomatic.threading.ParserWorker.run(ParserWorker.java:56)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException: Unexpected end of ZLIB input stream
    at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:240)
    at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
    at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117)
    at org.usadellab.trimmomatic.util.ConcatGZIPInputStream.read(ConcatGZIPInputStream.java:73)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.fill(BufferedReader.java:161)
    at java.io.BufferedReader.readLine(BufferedReader.java:324)
    at java.io.BufferedReader.readLine(BufferedReader.java:389)
    at org.usadellab.trimmomatic.fastq.FastqParser.parseOne(FastqParser.java:85)
    at org.usadellab.trimmomatic.fastq.FastqParser.next(FastqParser.java:179)
    at org.usadellab.trimmomatic.threading.ParserWorker.run(ParserWorker.java:42)
    ... 1 more

Terminal Message:

Input Read Pairs: 62991000 Both Surviving: 23166679 (36.78%) Forward Only Surviving: 8892038 (14.12%) Reverse Only Surviving: 6210627 (9.86%) Dropped: 24721656 (39.25%)
TrimmomaticPE: Completed successfully

This data had sequencing problems, the low % surviving isn't surprising

Fastq Output (example):

@ERR407508.7847495 HWI-ST143:472:D0JDUACXX:4:1208:5586:161136/1
GTAACAGCCGTTTGACGTAACAAACTGAACTTGAGGTTAACTCCCGGATCATCCCGGGTCACGCGATGACAACGTAAGCTAATTTGATCCGTTGATACAGA
+
@?@DF?DDHFHHFIHIGHIIJJEHJIIEHIGJJEGAD@FHHGJGHIHBEHIIGIGGEG9=ACCBBDBDCCDDDD?8??CACDDCDADC@C?<ABDDDDDCC

[lieven.sterck]

is that a head or a tail of your fastq file?

I think the ZLIB error message is triggered by the fact that the streams are not opened or closed correctly, so there might be something missing at the end of your fastq file.

[James Reeve]

That is the head. I checked the tail, it also looks normal.

[GenoMax]

Have you checked the integrity of your fastq file? It may be corrupt.validateFiles from Jim Kent's utilities should work.

[Pierre Lindenbaum]

test your fastq files:

gunzip -t your.fastq.gz && echo "VALID"

[James Reeve]

Thanks Pierre, this is a useful command for the future. In this case it was VALID

[GenoMax]

Please use validateFiles to ensure you have a valid fastq file with no broken records. Error could be anywhere in the file. You can also try fastqvalidator for this purpose.

[James Reeve]

It took a while to download, but i managed to run fastqvalidator. No errors showed up.

I wasn't able to get validateFiles to work. The command keeps telling me cannot execute binary file.

[GenoMax]

Did you add execute permission to validateFiles by doing chmod a+x validateFiles? I had linked linux version. If you are using macOS then you should get this file. Did you try running trimmomatic again? Perhaps the first time was just a stray error, if the file checks out otherwise.

[James Reeve]

Yes I found an error with validateFiles. I accidentally used the output file, that's why no error showed up before before. Thanks for the help.

 gzip: (stdin): unexpected end of file    
 Error [file=Us_L1-1.fastq.gz, line=251964208]: quality not as long as sequence (101 bases) [BCCFFDD>>DFHFIIJJJJJJJJJJIJJJJGIJIJJJIJJIJJJIJJJJJIIIJJJJDEHJJIJHHHHHBDFDDEB@DDDDDDDBDDDDDDCDCC@:>CD]
Aborting .. found 1 error

[GenoMax]

If you are willing, I suggest that you give bbduk.sh from BBMap suite a try and see if that is able to complete the trimming. There is a guide here.

[h.mon]

Maybe the fastq.gz files are corrupted? Did you run them through FastQC? Did you try zcat | head and zcat | tail on those files?

[James Reeve]

Just tried it. Running FastQC generates normal results and zcat produces the same output as the gunzip file.

[lieven.sterck]

How exactly do you run the original command? Do you run it in your terminal or send it to a compute cluster?

[James Reeve]

I ran it through terminal, in a while loop that ran through 30 different files. All the other files ran without error.