samtools: writing to standard output failed: Broken pipe

Question

[mcff23]

Hi everyone!

I'm getting this error from tophat:

samtools: writing to standard output failed: Broken pipe

Log file:

[2014-09-15 16:31:23] Beginning TopHat run (v2.0.12)
-----------------------------------------------
[2014-09-15 16:31:23] Checking for Bowtie
                  Bowtie version:        2.2.3.0
[2014-09-15 16:31:23] Checking for Samtools
[2014-09-15 16:31:23] Checking for Bowtie index files (genome)..
[2014-09-15 16:31:23] Checking for reference FASTA file
[2014-09-15 16:31:23] Generating SAM header for 
[2014-09-15 16:31:23] Reading known junctions from GTF file
[2014-09-15 16:31:27] Preparing reads
         left reads: min. length=101, max. length=101, 99994 kept reads (6 discarded)
        right reads: min. length=101, max. length=101, 99455 kept reads (545 discarded)
[2014-09-15 16:31:32] Building transcriptome data files test_LC1.THout/tmp/Lj2.5_gene_models
[2014-09-15 16:31:38] Building Bowtie index from Lj2.5_gene_models.fa
[2014-09-15 16:32:07] Mapping left_kept_reads to transcriptome Lj2.5_gene_models with Bowtie2 
[2014-09-15 16:32:13] Mapping right_kept_reads to transcriptome Lj2.5_gene_models with Bowtie2 
[2014-09-15 16:32:18] Resuming TopHat pipeline with unmapped reads
samtools: writing to standard output failed: Broken pipe
samtools: error closing standard output: -1
samtools: writing to standard output failed: Broken pipe
samtools: error closing standard output: -1
[2014-09-15 16:32:18] Mapping left_kept_reads.m2g_um to genome Bowtie2Index with Bowtie2 
samtools: writing to standard output failed: Broken pipe
samtools: error closing standard output: -1
[2014-09-15 16:32:28] Mapping left_kept_reads.m2g_um_seg1 to genome Bowtie2Index with Bowtie2 (1/4)
[2014-09-15 16:32:31] Mapping left_kept_reads.m2g_um_seg2 to genome Bowtie2Index with Bowtie2 (2/4)
[2014-09-15 16:32:33] Mapping left_kept_reads.m2g_um_seg3 to genome Bowtie2Index with Bowtie2 (3/4)
[2014-09-15 16:32:36] Mapping left_kept_reads.m2g_um_seg4 to genome Bowtie2Index with Bowtie2 (4/4)
samtools: writing to standard output failed: Broken pipe
samtools: error closing standard output: -1
[2014-09-15 16:32:41] Mapping right_kept_reads.m2g_um to genome Bowtie2Index with Bowtie2 
samtools: writing to standard output failed: Broken pipe
samtools: error closing standard output: -1
[2014-09-15 16:32:50] Mapping right_kept_reads.m2g_um_seg1 to genome Bowtie2Index with Bowtie2 (1/4)
[2014-09-15 16:32:53] Mapping right_kept_reads.m2g_um_seg2 to genome Bowtie2Index with Bowtie2 (2/4)
[2014-09-15 16:32:56] Mapping right_kept_reads.m2g_um_seg3 to genome Bowtie2Index with Bowtie2 (3/4)
[2014-09-15 16:32:59] Mapping right_kept_reads.m2g_um_seg4 to genome Bowtie2Index with Bowtie2 (4/4)
[2014-09-15 16:33:03] Searching for junctions via segment mapping
[2014-09-15 16:33:18] Retrieving sequences for splices
[2014-09-15 16:33:31] Indexing splices
Building a SMALL index
samtools: writing to standard output failed: Broken pipe
samtools: error closing standard output: -1
[2014-09-15 16:33:36] Mapping left_kept_reads.m2g_um_seg1 to genome segment_juncs with Bowtie2 (1/4)
[2014-09-15 16:33:38] Mapping left_kept_reads.m2g_um_seg2 to genome segment_juncs with Bowtie2 (2/4)
[2014-09-15 16:33:39] Mapping left_kept_reads.m2g_um_seg3 to genome segment_juncs with Bowtie2 (3/4)
[2014-09-15 16:33:41] Mapping left_kept_reads.m2g_um_seg4 to genome segment_juncs with Bowtie2 (4/4)
[2014-09-15 16:33:42] Joining segment hits
samtools: writing to standard output failed: Broken pipe
samtools: error closing standard output: -1
samtools: writing to standard output failed: Broken pipe

and so on..

I couldn't find any specific solution...

Thanks in advance!

MF

[Jeff Hussmann]

I think that I have tracked the issue down to the following code in tophat.py:

samtools_view_cmd = ["samtools", "view", filepath]
samtools_view = subprocess.Popen(samtools_view_cmd, stdout=subprocess.PIPE)
head_cmd = ["head", "-1"]
head = subprocess.Popen(head_cmd, stdin=samtools_view.stdout, stdout=subprocess.PIPE)
samtools_view.stdout.close() # as per http://bugs.python.org/issue7678
output = head.communicate()[0][:-1]

The issue appears to be that python's handling of SIGPIPEs in this kind of pipeline is nonstandard - see http://www.chiark.greenend.org.uk/ucgi/~cjwatson/blosxom/2009-07-02-python-sigpipe.html - and that the newer version of samtools cares about this, while the old version didn't.

Installing the backport of python3k's subprocess at http://code.google.com/p/python-subprocess32/ and changing the above code to use this version of subprocess gets rid of the error messages.

[Nicolas Rosewick]

Could you put your tophat command you used? Do you have the write permission in the tophat output directory? And also samtools version? Thanks

[mcff23]

HI NicoBxl!

command:

tophat -p 8 -G annotation -o sample.THout bowtie2_index R1.fastq.gz R2.fastq.gz

Yes I have write permissions in the output directory. The samtools version is the latest 1.0

Thanks!

MF

[Devon Ryan]

How much memory do you have on the computer running this? Also, have you tried an older version of samtools? Tophat2 doesn't natively support the most recent version.

[mcff23]

I have 32 Gb of RAM and the 80% is free.

I patched (https://groups.google.com/forum/#!topic/tuxedo-tools-users/BVNDDmInU0A) tophat for samtools 1.0 version support.

[Devon Ryan]

With that much RAM you'd be better off using STAR. Tophat is painfully slow and mildly annoying to debug.

[Istvan Albert]

I would not muck around with unofficial patches, so many things can also break, just use the supported samtools until the main trunk supports it natively

Invoke the new samtools separately in your own work

[spiderdijon]

Check that your disk-space isn't close to full

[Shicheng Guo]

When I submit Bismark jobs in PBS system, I did get some similar error report. However, every result have been successfully obtained. Even that the error log file will show you:

bismark_methylation_extractor --single-end --bedGraph --cutoff 5 --multicore 3 --ignore 3 --ignore_3prime 3 --gzip --buffer_size 4G --zero_based --comprehensive --output ../methyfreq/ CTR154_trimmed.fq.gz_bismark_bt2.bam
Finished BedGraph conversion ...
gzip: stdout: Broken pipe

Don't care about them. I think it is the problem of PBS system.

[cjgunase]

the same thing happened to me as well. This is the end of fhe PBS output file. but I got the coverage file.

Finished generating genome-wide cytosine report

samtools view: writing to standard output failed: Broken pipe
samtools view: error closing standard output: -1

gzip: stdout: Broken pipe

Answer 1

[mcff23]

The solution for this error was using an older version of samtools like Devon Ryan said. I though the patch was working fine and it wasn't.

With Samtools 0.1.19.0 it worked just fine!!

Thanks a lot guys!

[xieshaojun0621]

This error may also happen when using bam2cfg.pl in BreakDancer.

The solution was the same: use an older version.

[Devon Ryan]

Thanks for reporting back and also mentioning this on the tophat forum.