How to export stdout of bash script to a text file

Question

[Prangan]

Hello! I am working on a computational pipeline implemented as a bash script. I want a summary file to be created on running the script containing the stdout of the script. I have come across

tee

but my pipeline requires a terminal output as well as a summary text file containing the stdout of the pipeline run.

Also, I have

zenity

in my pipeline, if that is of any consequence. Appreciating all the help! Thanks

Answer 1

[ATpoint]

There are many ways to do this. You could send terminal output to stdout and save any summary logs first to some sort of variable or array and then print it later upon completion of the pipeline. Or make screen messages sent to stdout and the rest to stderr, many options here. The usual advise, don't use bash scripts for pipelines, learn a proper workflow manager unless the task is trivially simple. It is cumbersome at first, but a WFmanager makes sense in the longterm. For more info you would need to give any examples.

[Prangan]

Thanks for the advice! I am currently working on nextflow wrapping of the script. Coming back to my problem, I am attaching an stdout snippet below:

Running FASTP
fastp --thread=50 --length_required=10 --qualified_quality_phred=32 --in1=/home/cluster/nath/run6/query/fastq/SRR10333577_1.fastq.gz --in2=/home/cluster/nath/run6/query/fastq/SRR10333577_2.fastq.gz --out1=SRR10333577_1_trimmed.fastq.gz --out2=SRR10333577_2_trimmed.fastq.gz --json=SRR10333577_1.json --html=SRR10333577_1.html 
fastp v0.20.0, time used: 231 seconds
Running FASTQC
Started analysis of SRR10333577_1_trimmed.fastq.gz
Started analysis of SRR10333577_2_trimmed.fastq.gz
Approx 5% complete for SRR10333577_1_trimmed.fastq.gz
Approx 5% complete for SRR10333577_2_trimmed.fastq.gz
Approx 10% complete for SRR10333577_1_trimmed.fastq.gz
Approx 10% complete for SRR10333577_2_trimmed.fastq.gz
Approx 15% complete for SRR10333577_1_trimmed.fastq.gz
Approx 15% complete for SRR10333577_2_trimmed.fastq.gz
Approx 20% complete for SRR10333577_1_trimmed.fastq.gz
Approx 20% complete for SRR10333577_2_trimmed.fastq.gz
Approx 25% complete for SRR10333577_1_trimmed.fastq.gz
Approx 25% complete for SRR10333577_2_trimmed.fastq.gz
Approx 30% complete for SRR10333577_1_trimmed.fastq.gz
Approx 30% complete for SRR10333577_2_trimmed.fastq.gz
Approx 35% complete for SRR10333577_1_trimmed.fastq.gz
Approx 35% complete for SRR10333577_2_trimmed.fastq.gz
Approx 40% complete for SRR10333577_1_trimmed.fastq.gz
Approx 40% complete for SRR10333577_2_trimmed.fastq.gz
Approx 45% complete for SRR10333577_1_trimmed.fastq.gz
Approx 45% complete for SRR10333577_2_trimmed.fastq.gz
Approx 50% complete for SRR10333577_1_trimmed.fastq.gz
Approx 50% complete for SRR10333577_2_trimmed.fastq.gz
Approx 55% complete for SRR10333577_1_trimmed.fastq.gz
Approx 55% complete for SRR10333577_2_trimmed.fastq.gz
Approx 60% complete for SRR10333577_1_trimmed.fastq.gz
Approx 60% complete for SRR10333577_2_trimmed.fastq.gz
Approx 65% complete for SRR10333577_1_trimmed.fastq.gz
Approx 65% complete for SRR10333577_2_trimmed.fastq.gz
Approx 70% complete for SRR10333577_1_trimmed.fastq.gz
Approx 70% complete for SRR10333577_2_trimmed.fastq.gz
Approx 75% complete for SRR10333577_1_trimmed.fastq.gz
Approx 75% complete for SRR10333577_2_trimmed.fastq.gz
Approx 80% complete for SRR10333577_1_trimmed.fastq.gz
Approx 80% complete for SRR10333577_2_trimmed.fastq.gz
Approx 85% complete for SRR10333577_1_trimmed.fastq.gz
Approx 85% complete for SRR10333577_2_trimmed.fastq.gz
Approx 90% complete for SRR10333577_1_trimmed.fastq.gz
Approx 90% complete for SRR10333577_2_trimmed.fastq.gz
Approx 95% complete for SRR10333577_1_trimmed.fastq.gz
Approx 95% complete for SRR10333577_2_trimmed.fastq.gz
Running HISAT2
Settings:
  Output files: "index.*.ht2"
  Line rate: 6 (line is 64 bytes)
  Lines per side: 1 (side is 64 bytes)
  Offset rate: 4 (one in 16)
  FTable chars: 10
  Strings: unpacked
  Local offset rate: 3 (one in 8)
  Local fTable chars: 6
  Local sequence length: 57344
  Local sequence overlap between two consecutive indexes: 1024
  Endianness: little
  Actual local endianness: little
  Sanity checking: disabled
  Assertions: disabled
  Random seed: 0
  Sizeofs: void*:8, int:4, long:8, size_t:8
Input files DNA, FASTA:
  /home/cluster/nath/run6/query/ref_fa//ref.fa
Reading reference sizes
  Time reading reference sizes: 00:00:25
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
  Time to join reference sequences: 00:00:15
  Time to read SNPs and splice sites: 00:00:00
Using parameters --bmax 582334929 --dcv 1024
  Doing ahead-of-time memory usage test
  Passed!  Constructing with these parameters: --bmax 582334929 --dcv 1024
Constructing suffix-array element generator
Converting suffix-array elements to index image
Allocating ftab, absorbFtab
Entering GFM loop

I want all the stdout (including the echo points) to be saved to an external text file. I would be much obliged to get a detailed or semi-detailed idea on how to do that. Thanks!

[ATpoint]

With Nextflow that would all end up in the .nextflow.log file of the respective work directory by default. You can of course try and catch it in a file with either > for stdout or 2> for stderr, depending to which channel the tools sends it, like:

tool do_something -p 1 -t 2 -e 3 (...) > stdout.log 2> stderr.log

[Prangan]

I guess that should do it. Thanks for the help. Just one more thing, could you refer me to any other resource that can help me wrap my script with nextflow? Other than the nextflow manual, I mean. Appreciate all the help. Thanks again!

[Pierre Lindenbaum]

ust one more thing, could you refer me to any other resource that can help me wrap my script with nextflow?

show us what you tried, ask this as a new question.