Picard SamToFastq error

Question

[songtree05]

Trying to convert a .bam file to paired .fastq files using Picard SamToFastq 2.5.0 and hitting an error part of the way through an early chromosome. First time I ran this it ran for 12 minutes and got to the beginning of chr4, second time, shown below it only got into chr3 before hitting a write error

Running:

picard SamToFastq INPUT=filename FASTQ=filename_1.fastq SECOND_END_FASTQ=filename_2.fastq UNPAIRED_FASTQ=filename_unpaired.fastq VALIDATION_STRINGENCY=LENIENT

It ran for about 10 minutes:

INFO    2018-07-17 12:08:39 SamToFastq  Processed     1,000,000 records.  Elapsed time: 00:00:28s.  Time for last 1,000,000:   28s.  Last read position: chr1:22,140,913

INFO    2018-07-17 12:09:19 SamToFastq  Processed     2,000,000 records.  Elapsed time: 00:01:08s.  Time for last 1,000,000:   40s.  Last read position: chr1:43,217,951

INFO    2018-07-17 12:09:57 SamToFastq  Processed     3,000,000 records.  Elapsed time: 00:01:47s.  Time for last 1,000,000:   38s.  Last read position: chr1:76,877,805

INFO    2018-07-17 12:10:33 SamToFastq  Processed     4,000,000 records.  Elapsed time: 00:02:23s.  Time for last 1,000,000:   35s.  Last read position: chr1:117,617,758

INFO    2018-07-17 12:11:10 SamToFastq  Processed     5,000,000 records.  Elapsed time: 00:02:59s.  Time for last 1,000,000:   36s.  Last read position: chr1:156,028,112

INFO    2018-07-17 12:11:45 SamToFastq  Processed     6,000,000 records.  Elapsed time: 00:03:34s.  Time for last 1,000,000:   35s.  Last read position: chr1:179,989,267

INFO    2018-07-17 12:12:23 SamToFastq  Processed     7,000,000 records.  Elapsed time: 00:04:12s.  Time for last 1,000,000:   37s.  Last read position: chr1:210,944,981

INFO    2018-07-17 12:12:59 SamToFastq  Processed     8,000,000 records.  Elapsed time: 00:04:48s.  Time for last 1,000,000:   36s.  Last read position: chr1:244,582,046

INFO    2018-07-17 12:13:38 SamToFastq  Processed     9,000,000 records.  Elapsed time: 00:05:27s.  Time for last 1,000,000:   38s.  Last read position: chr2:30,965,101

INFO    2018-07-17 12:14:23 SamToFastq  Processed    10,000,000 records.  Elapsed time: 00:06:13s.  Time for last 1,000,000:   45s.  Last read position: chr2:69,646,397

INFO    2018-07-17 12:15:01 SamToFastq  Processed    11,000,000 records.  Elapsed time: 00:06:50s.  Time for last 1,000,000:   37s.  Last read position: chr2:107,123,340

INFO    2018-07-17 12:15:40 SamToFastq  Processed    12,000,000 records.  Elapsed time: 00:07:29s.  Time for last 1,000,000:   38s.  Last read position: chr2:152,483,416

INFO    2018-07-17 12:16:15 SamToFastq  Processed    13,000,000 records.  Elapsed time: 00:08:04s.  Time for last 1,000,000:   35s.  Last read position: chr2:179,526,589

INFO    2018-07-17 12:16:49 SamToFastq  Processed    14,000,000 records.  Elapsed time: 00:08:39s.  Time for last 1,000,000:   34s.  Last read position: chr2:212,566,734

INFO    2018-07-17 12:17:24 SamToFastq  Processed    15,000,000 records.  Elapsed time: 00:09:13s.  Time for last 1,000,000:   34s.  Last read position: chr2:240,935,555

INFO    2018-07-17 12:18:02 SamToFastq  Processed    16,000,000 records.  Elapsed time: 00:09:51s.  Time for last 1,000,000:   37s.  Last read position: chr3:37,791,987

[Tue Jul 17 12:18:02 EDT 2018] picard.sam.SamToFastq done. Elapsed time: 9.88 minutes.

Runtime.totalMemory()=466288640
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" htsjdk.samtools.SAMException: Error in writing fastq file filename_1.fastq
    at htsjdk.samtools.fastq.BasicFastqWriter.write(BasicFastqWriter.java:68)
    at picard.sam.SamToFastq.writeRecord(SamToFastq.java:350)
    at picard.sam.SamToFastq.doWork(SamToFastq.java:195)
    at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:208)
    at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
    at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)

[ATpoint]

Keep in mind that if this is a coordinate-sorted BAM file, you should sort it first by name to restore the original order as the reads came from the sequences (= random order towards genomic coordinates). Aligners such as BWA mem expect randomly sorted fastq files as input. Has to do with how they calculate the insert sizes. Agree with Pierre to use samtools for this. Might be a matter of taste but I hate these java utilities from Broad, because they are overloaded with options and often prompt error messages like the above one that I cannot really get my head around (probably just a lack of java knowledge on my side):

samtools sort -n in.bam | samtools fastq -0 out_unpaired.fq.gz -1 out_1.fq.gz -2 out_2.fq.gz

Answer 1

[Pierre Lindenbaum]

it's an I/O error.

check you have the right to write in the current directory.

check you have the right to write a large amount of data in the current directory.

try an alternative solution samtools fastq http://www.htslib.org/doc/samtools.html

[GenoMax]

You could also use reformat.sh in=your.sam out1=R1.fq.gz out2=R2.fq.gz (BBMap suite) as another option.