Naive BASH question on top to bottom programming regarding my alignment script
1
0
Entering edit mode
6.8 years ago

I'm confused as to why the two bowtie lines are running at the same time? I was under the impression that everything in unix is run top to bottom and the second bowtie line wouldn't start running until the first is completed. The reason I ask is these commands are running simultaneously and I'm having memory issues of sorting two large bam files at the same time.. I'm assuming it has something to do with the piping but I am confused. Script below

(I have changed the sample names per privacy)

!/bin/bash
echo BOWTIE LOG > ../BAM/bowtie_log.txt
mkdir ../BAM/
bowtie2 --very-sensitive -x /PATH/TO/genome -X 2000 2>> ../BAM/bowtie_log.txt -p 1 -1 trimmed/Sample1_R1_001_val_1.fq.gz -2 trimmed/Sample1_R2_001_val_2.fq.gz | samtools view -Sb - | samtools sort - -o ../BAM/Sample1.bam
bowtie2 --very-sensitive -x /PATH/TO/genome -X 2000 2>> ../BAM/bowtie_log.txt -p 1 -1 trimmed/Sample2_R1_001_val_1.fq.gz -2 trimmed/Sample2_R2_001_val_2.fq.gz | samtools view -Sb - | samtools sort - -o ../BAM/Sample2.bam
Unix bash • 2.1k views
ADD COMMENT
1
Entering edit mode

(Edited my original comment based on @Jean-Karim's comment and some other searches): It may be the pipes in the command that are backgrounding something and allowing both jobs to run. There are no logical/control operators.

Adding { } around the entire command (as suggested by @steve below) may prevent that backgrounding from happening (if it works, my assumption).

(see this, specifically A.2 logical operators).

ADD REPLY
0
Entering edit mode

Wow learn something new about bash everyday. Thanks for the input

ADD REPLY
0
Entering edit mode

I don't follow this. This has to be more subtle than that. The newline is a terminator and the shell should wait for the command to complete before continuing on the next one.

I think the OP does not have a new line at all and the text wraps around and shows as if there were a newline.

ADD REPLY
0
Entering edit mode

I added a comment to Jean's answer? Does this bg have this effect?

ADD REPLY
0
Entering edit mode

does the same thing happen if you write it as { bowtie2 .... | samtools ... ; } && { bowtie2 .... | samtools ... ; } ?

ADD REPLY
0
Entering edit mode

What is making you think that both are running concurrently?

ADD REPLY
0
Entering edit mode

Reason I thought that was because Sample2 tmp merge files were being made before Sample1 was merged and got a memory error

ADD REPLY
2
Entering edit mode
6.8 years ago

Unless told otherwise (e.g. with &), bash processes lines sequentially waiting for the first to finish before starting the next one. In the case here, if the two bowtie2 commands run concurrently, something in the first line must have forked/backgrounded itself.

ADD COMMENT
0
Entering edit mode

Oh this is a good point. I did start the command and then put it in the background by control-z and bg, would that have an effect??

ADD REPLY
1
Entering edit mode

ok, case closed. with that you've made the programs run in parallel - I also moved the comment to answer.

ADD REPLY
0
Entering edit mode

Got it. Sorry for the confusion and not saying everything in my original post

ADD REPLY

Login before adding your answer.

Traffic: 1936 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6