Bash loop for bbduk

Question

[avelarbio46]

Hello Everyone! I'm trying to run a loop to trim my rnaseq reads in bbduk. However, I can't seem to input files from a specific directory nor output them to a directory. Probably this is very easy to solve. I'm currently using:

for i in `ls -1 /home/gabriel.gama/Dados_CD_genomics/TrueSeq_dezembro/*_1.fq.gz | sed 's/_1.fq.gz//'`
do
bbduk.sh -Xmx1g in1=$i\_1.fq.gz in2=$i\_2.fq.gz out1=/home/gabriel.gama/Análises/Teste1/bbduk/$i\_clean_1.fq.gz out2=/home/gabriel.gama/Análises/Teste1/bbduk/$i\_clean_2.fq.gz ref=/home/gabriel.gama/bbduk/bbmap/resources/adapters.fa ktrim=r k=23 mink=11 hdist=1 tpe tbo qtrim=r trimq=10 maq=10 
done

I'm not getting the sample names as output, even tough the command run. I'm getting the absolute path to file names as output

Maybe I should use something such as

for a in 'basename $i'

to get the basename of the file, and then reference it as such:

out1=/home/gabriel.gama/Análises/Teste1/bbduk/$a\_clean_1.fq.gz 

[rpolicastro]

what error are you getting?

[avelarbio46]

I can't get the "samplename.fq" as output, just absolute paths

Answer 1

[GenoMax]

Please do not use spaces in folder names. It is just simpler to use a _ when you feel like using a space. It would be better name my machine to my_machine.

There can be no spaces in bbduk.sh options. You seem to have a space between ref= and the directory after it.

Try this:

 for i in `ls -1 /home/gabriel.gama/Dados_CD_genomics/TrueSeq_dezembro/*_1.fq.gz`; \
    do dname=$(dirname ${i}); name=$(basename ${i} _1.fq.gz); \
    bbduk.sh -Xmx1g in1=${dname}/${name}_1.fq.gz in2=${dname}/${name}_2.fq.gz \
    out1=/home/gabriel.gama/Análises/Teste1/bbduk/${name}_clean_1.fq.gz out2=/home/gabriel.gama/Análises/Teste1/bbduk/${name}_clean_2.fq.gz \
    ref=/home/gabriel.gama/bbduk/bbmap/resources/adapters.fa ktrim=r k=23 mink=11 hdist=1 tpe tbo qtrim=r trimq=10 maq=10 ; 
done

[avelarbio46]

I removed spaces where possible, but I still can't reference to the sample name, just absolute path. Because of that I can't output the fastq correctly :(

Answer 2

[Mensur Dlakic]

Try setting up basename in your for command:

for i in `ls -1 /home/gabriel.gama/Dados_CD_genomics/TrueSeq_dezembro/*_1.fq.gz | sed 's/_1.fq.gz//' | xargs -i basename {}`

[avelarbio46]

This makes "$i" become basename, but then I can't use it to define the absolute path in

bbduk.sh -Xmx1g in1=$i\_1.fq.gz

[Mensur Dlakic]

Come on now, you can't expect us to change every single line of code. It helps if you put some effort of your own, which here would be adding a fixed directory part to in1 and in2:

bbduk.sh -Xmx1g in1=/home/gabriel.gama/Dados_CD_genomics/TrueSeq_dezembro/$i\_1.fq.gz in2=/home/gabriel.gama/Dados_CD_genomics/TrueSeq_dezembro/$i\_2.fq.gz out1=/home/gabriel.gama/Análises/Teste1/bbduk/$i\_clean_1.fq.gz out2=/home/gabriel.gama/Análises/Teste1/bbduk/$i\_clean_2.fq.gz ref=/home/gabriel.gama/bbduk/bbmap/resources/adapters.fa ktrim=r k=23 mink=11 hdist=1 tpe tbo qtrim=r trimq=10 maq=10 

[avelarbio46]

I agree with you! I'm really trying to learn about loops and more complex functions, but I couldn't get your answer to work before the edits!

[GenoMax]

See the modified version I added to my answer above.

Answer 3

[cpad0112]

I think code is a little bit complex as you are getting sample read name (ls), converting to the dir name (sed) and then adding read information (in loop). Please check the following code and see if you can improve on it. I have removed the paths, rephrased the code and see if it makes sense. (please add paths wherever applicable). This code works when wherever the reads and adapters are kept in the same directory:

with bash loop:

$ for i in $(ls  *_1.fq.gz);                                                                                                                                                          
do echo bbduk.sh -Xmx1g in1=$i in2=${i/_1/_2} out1=./bbduk_results/${i/_1/_clean_1} out2=./bbduk_results/${i/_1/_clean_2} ref=adapters.fa ktrim=r k=23 mink=11 hdist=1 tpe tbo qtrim=r trimq=10 maq=10;
done

bbduk.sh -Xmx1g in1=sample1_1.fq.gz in2=sample1_2.fq.gz out1=./bbduk_results/sample1_clean_1.fq.gz out2=./bbduk_results/sample1_clean_2.fq.gz ref=adapters.fa ktrim=r k=23 mink=11 hdist=1 tpe tbo qtrim=r trimq=10 maq=10
bbduk.sh -Xmx1g in1=sample2_1.fq.gz in2=sample2_2.fq.gz out1=./bbduk_results/sample2_clean_1.fq.gz out2=./bbduk_results/sample2_clean_2.fq.gz ref=adapters.fa ktrim=r k=23 mink=11 hdist=1 tpe tbo qtrim=r trimq=10 maq=10

Please consider using find instead of ls.

with parallel:

$ parallel --plus --dry-run bbduk.sh  -Xmx1g in1={} in2={=s/_1/_2/=} out1=./bbduk_results/{=s/_1/_clean_1/=} out2=./bbduk_results/{=s/_1/_clean_2/=} ref=adapters.fa ktrim=r k=23 mink=11 hdist=1 tpe tbo qtrim=r trimq=10 maq=10 ::: *_1.fq.gz

bbduk.sh -Xmx1g in1=sample1_1.fq.gz in2=sample1_2.fq.gz out1=./bbduk_results/sample1_clean_1.fq.gz out2=./bbduk_results/sample1_clean_2.fq.gz ref=adapters.fa ktrim=r k=23 mink=11 hdist=1 tpe tbo qtrim=r trimq=10 maq=10
bbduk.sh -Xmx1g in1=sample2_1.fq.gz in2=sample2_2.fq.gz out1=./bbduk_results/sample2_clean_1.fq.gz out2=./bbduk_results/sample2_clean_2.fq.gz ref=adapters.fa ktrim=r k=23 mink=11 hdist=1 tpe tbo qtrim=r trimq=10 maq=10

Please also consider using snakemake or any other framework.

Answer 4

[avelarbio46]

After a lot of help, I learned one the fastest and easiest way to do so in a for loop. The thing is, manipulating $i is better, such as:

for i in `ls -1 /home/gabriel.gama/Dados_CD_genomics/TrueSeq_dezembro/*_1.fq.gz | sed 's/_1.fq.gz//'` 
do
bbduk.sh -Xmx1g in1=$i\_1.fq.gz in2=$i\_2.fq.gz out1=/home/gabriel.gama/Análises/Teste1/bbduk/${i##*/}\_clean_1.fq.gz out2=/home/gabriel.gama/Análises/Teste1/bbduk/${i##*/}\_clean_2.fq.gz ref=/home/gabriel.gama/bbduk/bbmap/resources/adapters.fa ktrim=r k=23 mink=11 hdist=1 tpe tbo qtrim=r trimq=10 maq=10 
done

The term

${i##*/}

gives the basename of the directory!

Thanks https://stackoverflow.com/users/140750/william-pursell for the response