Save Output Into Text File, Whose Name Is A Variable

Question

[Bioscientist]

Hi I'm using SUN Grid Engine to submit jobs, written by shell script like this:

#!/bin/bash

#$ -pe single 24
#$ -V
#$ -cwd
#$ -o $HOME/sge_jobs_output/sge_job.$JOB_ID.out -j y
#$ -S /bin/bash
#$ -l mem_free=8G

cd $HOME/scratch/bwa2/

job_number=$SGE_TASK_ID
SAM_NAME_1=`head -"$job_number" master_list_1|tail -1`
/share/bin/bwa aln hg19_index "$SAM_NAME_1".recal.fastq.gz > "$SAM_NAME_1".sai

 #Do sth. here to save the output as text file. Here by output I don't mean
 # .sai file, but the error report like "10000000 sequences have been processed",
 #And text file name is SAM_NAME_1, which is a variable

Maybe I didn't make quite clear. As shown as the end of the script, I want to put some command so that the error report can be saved as .txt file. How can I do that? (I know we can use > .txt at prompt; but here the .txt file name is a variable, so I must put command within script)

WHat I want is similar to $HOME/sgejobsoutput/sgejob.$JOBID.out -j y; but I want name as SAMNAME instead of JOBID. Thanks!

SO actually this turned out as a bioinformatic problem. :)

Edit:

I tried: #$ -e $HOME/sge_jobs_output/sge_job.$SAM_NAME_1.txt But it still doesn't work..$SAMNAME1 part doesn't act as variable..

Is there any other way, which does NOT rely on this SGE flag? thx

[brentp]

I think this isn't really bioinformatics related. but you likely want to use ${SAM_NAME_1}.recal.fastq.gz and ${SAM_NAME_1}.sai

[Bioscientist]

Yeah, I know it's NOT bioinformatics-related; I post on stack overflow, but got no reply. So how should I do to save output as text file here? thanks.

[biobot 0.0.77.a.1099]

Here's a fishing rod: the Advanced Bash-Scripting Guide http://www.tldp.org/LDP/abs/html/ . See Chapter 4. Introduction to Variables and Parameters.

Answer 1

[Mitch Bekritsky]

The way to redirect STDERR to a file in SGE is by using the -e flag. The trick for your program is to substitute the SAM file name for the job name in the argument for -e. For you, all you should have to add is

 #! -e "${HOME}/sge_jobs_output/sge_job.${SAM_NAME}.txt"

I haven't tested this out, but I know for certain that redirecting error output from SGE is with -e. Naming the file should be the easy part (I hope...).

Hope that's useful!

[Bilouweb]

Obviously !! I did not used sge for a long time and forgot this flag ! +1

[Bioscientist]

hi, seems it still doesn't work. It's showing 'sge_job.$SAM_NAME.txt', seems SAM_NAME doesn't work as variable...

[Bioscientist]

I tried: #$ -e $HOME/sge_jobs_output/sge_job.$SAM_NAME_1.txt But it still doesn't work..$SAM_NAME_1 part doesn't act as variable..

[Mitch Bekritsky]

When I set error output using the -e command, I set the filename at the command line (e.g. qsub ... -e [error file name] ...), and I know it will work there, not sure if it will work in the script, but I think it should. Do you know if $SAM_NAME_1 has the the value you want it to? That would be the first place I would check. Unfortunately, I'm away from my grid right now so I can't check it for you. Glad to help you trouble shoot though!

[Bioscientist]

hi, if I do in the command line as: "qsub XXX.sh -e $SAM_NAME_1" ,but here SAM_NAME_1 is a variable. Does that work?

[Mitch Bekritsky]

Hi bioscientist, that should work (it always does for me). Although be careful to name it right. In your case, it should be qsub XXX.sh -e $sge_job.{SAM_NAME_1}.txt. You probably had that in mind, but just in case...

[Mitch Bekritsky]

Hi bioscientist, that should work (it always does for me). Although be careful to name it right. In your case, it should be qsub XXX.sh -e "$sge_job.{SAM_NAME_1}.txt". You probably had that in mind, but just in case...

Answer 2

[Sean Davis]

As brentp notes, you just need to change "$SAM_NAME_1" to ${SAM_NAME_1}. You'll also need to continue the script with:

bwa samse ${SAM_NAME_1} ${SAM_NAME_1}.sai ${SAM_NAME_1}.recal.fastq.gz

I didn't test this, but you should get the idea. The important thing when using a batch system such as SGE is to make sure that your script runs outside of the batch system before trying to use the batch system.

[Sean Davis]

Yes. You'll need to specify the name directly in the shell script. There is no variable interpolation in the SGE directives.

[Bioscientist]

HI Sean, sorry I don't make clear what I want to do. I don't mean to make .sai file as .txt; but I want the error report like "how many sequences have been processed" saved as .txt. How can I do that?

[Sean Davis]

See the -e flag for SGE.

[Bioscientist]

I tried: #$ -e $HOME/sge_jobs_output/sge_job.$SAM_NAME_1.txt But it still doesn't work..$SAM_NAME_1 part doesn't act as variable...

[Bioscientist]

Yeah, but how can I redirect the error output in the shell script? ANy clues?