Gnu Parallel Quoting With Awk

Question

[tayebwajb]

As I try to get familiar with the GNU Parallel tutorial on quoting, I need some help to move faster.

The command I want to quote is

 parallel --dryrun "awk '{ print $0,"\t","{.}" }' {} > {}" ::: *foo.txt

Basically, I want to add the FILENAME as the last column of the tab-delimited file, for each input file.

Problem is that bash interpretes the above command as

awk '{ print /bin/bash,t,foo.txt }' foo.txt > foo.txt

Which doesn't work. So it needs appropriate quoting to work

Answer 1

[ole.tange]

I assume what you want to do is given the file foo.bar you want this run:

awk '{ print $0,"\t","foo" }' foo.bar > foo.bar

But that will not work, as you have foo.bar both as input an output. What will happen is that foo.bar will be 0 bytes long.

So you need a temporary file:

awk '{ print $0,"\t","foo" }' foo.bar > foo.bar.tmp; mv foo.bar.tmp foo.bar

To do that using parallel it might be easier to define a function:

doawk() {
   FILE=$1
   BASE=$2
   awk '{ print $0,"\t","'"$BASE"'" }' "$FILE" > "$FILE".tmp; mv "$FILE".tmp "$FILE"
}
export -f doawk
parallel doawk {} {.} ::: *.txt

This way you avoid double quoting the awk script.

If you want to avoid the function we are looking at:

parallel awk\ \'\{\ print\ \$0,\"\\t\",\"\'{.}\'\"\ \}\'\ {}\ \>\ {}.tmp\;\ mv\ {}.tmp\ {} ::: *.txt

Answer 2

[dapregi]

With no info about the output expected, this is what I think you are trying to do:

parallel "awk '{print \$0, "\t", FILENAME}' {} > {}.tmp; mv {}.tmp {}" ::: *input.txt

[tayebwajb]

I added information about the expected output.