Using VelvetOptimiser.pl within a forloop

Question

[c.e.chong]

Hi,

I am trying to run VelvetOptimiser on a number of paired end read files.

I have installed both velvet and velvetoptimiser via Conda. I wrote this forloop to loop through all of my read fliles:

for i in *_1_*.fastq; do VelvetOptimiser.pl -t 12 -s 19 -v --d ~/comp_genomics/velvet_optimiser_jo/${i%%_1_*.fastq}_assembly -f '-fastq -shortPaired $i ${i%%_1_*.fastq}_2_*.fastq'; done

I am however getting this error message:

Will run velvet optimiser with the following paramters:
Velveth parameter string:
    -fastq -shortPaired $i ${i%%_1_*.fastq}_2_*.fastq
Velveth start hash values:  19
Velveth end hash value:     191
Velveth hash step value:    2
Velvetg minimum coverage cutoff to use: 0

Read tracking for final assembly off. Jan 17 16:41:17 Beginning velveth runs. Jan 17 16:41:17       Running velveth with hash value: 19. velveth: Could not open _2_*.fastq: No such file or directory Velveth failed!  Response:1

I have tested the separate components of the command not in the forloop. So I know that separately the loop works as does the command options.

for i in *_1_*.fastq; do echo '-fastq -shortPaired' $i ${i%%_1_*.fastq}_2_*.fastq; done


VelvetOptimiser.pl -t 12 -s 19 -v --d ~/comp_genomics/velvet_optimiser_jo/bb_17198wB1_Sample_2_assembly -f '-shortPaired -fastq bb_17198wB1_Sample_2_1_trimmed_prinseq.fastq bb_17198wB1_Sample_2_2_trimmed_prinseq.fastq'

Is it possible to use VelvetOptimiser in a forloop? Is there a mistake that I'm missing?

I'd be very grateful for any help!

Thank you in advance!!!

[h.mon]

How come the output of

for i in *_1_*.fastq; do echo '-fastq -shortPaired' $i ${i%%_1_*.fastq}_2_*.fastq; done

is

VelvetOptimiser.pl -t 12 -s 19 -v --d ~/comp_genomics/velvet_optimiser_jo/bb_17198wB1_Sample_2_assembly -f '-shortPaired -fastq bb_17198wB1_Sample_2_1_trimmed_prinseq.fastq bb_17198wB1_Sample_2_2_trimmed_prinseq.fastq'

This is simply not possible.

In addition, explain your files naming convention, so we can better understand your problem. If all files are named {something}_trimmed_prinseq.fastq, I would use for i in *_1_trimmed_prinseq.fastq. If you pay attention, *_1_*.fastq will potentially match a file named bb_17198wB1_Sample_1_2_trimmed_prinseq.fastq, in case there is one.

[c.e.chong]

VelvetOptimiser.pl -t 12 -s 19 -v --d ~/comp_genomics/velvet_optimiser_jo/bb_17198wB1_Sample_2_assembly -f '-shortPaired -fastq bb_17198wB1_Sample_2_1_trimmed_prinseq.fastq bb_17198wB1_Sample_2_2_trimmed_prinseq.fastq'

is not the output of

for i in *_1_*.fastq; do echo '-fastq -shortPaired' $i ${i%%_1_*.fastq}_2_*.fastq; done

I was simply showing the two commands I ran to test both the velvet command itself and the the forloop.

An example of my file names is bb_17198wD7_Sample_52_1_trimmed_prinseq.fastq, I can take out the second wild card incase it would call the wrong file. But I haven't had this problem so far.

Thanks

Answer 1

[Joe]

By putting this section:

'-fastq -shortPaired $i ${i%%_1_*.fastq}_2_*.fastq'

in single quotes, you've hard quoted it, which means the variables aren't interpreted, its just treated as a string literal. You need to use ".

Also, Velvet is pretty old news at this point - is there a particular reason to use it in your case?

[c.e.chong]

Thanks for getting back to me!

I tried using " but this didn't work either. The input read files have to be within a ' to call Velveth.

I understand that Velvet is old news now, I was just trying to carry out some comparison assemblies as my group used to use velvet.

[Joe]

The fact that velvet spits out:

Will run velvet optimiser with the following paramters:
Velveth parameter string:
    -fastq -shortPaired $i ${i%%_1_*.fastq}_2_*.fastq

Tells me that your variables aren’t being expanded so that must be at least part of the problem. Do you get the same error when you replace the ‘ with “?

[c.e.chong]

When I replace the ' with " I get this error

Number of CPUs available: 64
Current free RAM: 496.545GB
Velvet OMP compiler setting: 1
Unknown option: shortpaired bb_17198wa9_sample_65_1_trimmed_prinseq.fastq bb_17198wa9_sample_65_2_*.fastq
Usage: /pub38/cchong/miniconda2/envs/velvet_env/bin/VelvetOptimiser.pl [options] -f 'velveth input line'

So this does call the correct files but this causes velveth not to run!

Thanks for your help!

[Joe]

Your filenames are expanding with spaces in which Velvet is interpreting as multiple unknown arguments.

If I do:

$ ls *.fastq.gz*
A_S1_L001_R1_001.fastq.gz  A_S1_L001_R2_001.fastq.gz

I have 2 dummy fastq files here, then:

for i in *_L001_* ; do echo "VelvetOptimiser.pl -t 12 -s 19 -v --d ~/comp_genomics/velvet_optimiser_jo/${i%%_1_*.fastq}_assembly -f -fastq -shortPaired $i ${i%%_1_*.fastq}_2_*.fastq" ; done

Gives:

VelvetOptimiser.pl -t 12 -s 19 -v --d ~/comp_genomics/velvet_optimiser_jo/A_S1_L001_R1_001.fastq.gz_assembly -f -fastq -shortPaired A_S1_L001_R1_001.fastq.gz A_S1_L001_R1_001.fastq.gz_2_*.fastq
VelvetOptimiser.pl -t 12 -s 19 -v --d ~/comp_genomics/velvet_optimiser_jo/A_S1_L001_R2_001.fastq.gz_assembly -f -fastq -shortPaired A_S1_L001_R2_001.fastq.gz A_S1_L001_R2_001.fastq.gz_2_*.fastq

There are multiple issues here. Firstly, you cannot just throw wildcards (*) in where ever you like. Particularly your second read file is specified totally wrong (${i%%_1_*.fastq}_2_*.fastq).

You need to start over and build the command up bit at a time, until you're certain that the variables expand correctly.

[c.e.chong]

Thank you all for your help I have resolved the issue now, using extra '.

'-shortPaired -fastq '$i' '${i%%_1_trimmed_prinseq.fastq}_2_trimmed_prinseq.fastq''