How to run Py-script 300 times with 3 new variables and a new output-file each time?

Question

[natasha.sernova]

Dear all,

My colleague wrote a Python2-script (there is no needed updated library in Python3 ).

To run it, I type

python script.py var1 var 2 ad1 ad2 ad3 ad4 var3 ad 5 ad6 ad7 ad8 ad9 –o var1_ouput.txt

var1, var2, var3 – are changed every run

ad1 ad2 ad3 ad4 ad 5 ad6 ad7 ad8 ad9 – are the same every run, but they are necessary, I cannot get rid of them

These constants (ad1 – ad9) interfere with the formalization of the problem in bash.

Please, give me a hint – how to deal with this problem? I wouldn’t like to run the script 300 times

manually… I need to have 300 output-files rather soon.

Thank you very much!

Natasha

[Martombo]

why would the constant arguments interfere with running a bash script?

[natasha.sernova]

It's a mixture of constant arguments and changing arguments.

ad3 ad4 var3 ad5 ad6

Fortunately var3 can be considered constant as well.

But in general case the solution looks more complicated.

[natasha.sernova]

Thank you very much, I didn't know so many different ways exist.

Answer 1

[finswimmer]

Hello natasha,

create a file where in every line you have the var (separated by tabs) that changes in every run, e.g.:

$ cat args
A   B   C
D   E   F

Then you can use gnu parallel to run your script with these arguments:

$ parallel --col-sep '\t' 'python script.py {1} {2} ad1 ad2 ad3 ad4 {3} ad5 ad6 ad7 ad8 ad9 –o {1}_ouput.txt' :::: args

fin swimmer

[Joe]

As an alternative, if you generate a text file with the full command as a string for every permutation you want, you can have parallel automatically spawn each process on one thread by looping through the file with $ parallel < my_commands.txt

Answer 2

[Carambakaracho]

Here's a very low tech quick and dirty solution using Excel (or anything similar): paste the columns in Excel much like @finswimmer suggested for the tab separated file as columns A through C. Then another column creates a bash script in row one:

="python script.py "&A1&" "&B1&" "&C1&"ad2 ad3 ad4 var3 ad 5 ad6 ad7 ad8 ad9 –o "&A1&"_ouput.txt;"

Finish by simple drag and drop to last row, copy paste to text file on your server and execute it. The only thing you should be aware of is that some editors might keep the Windows line endings, so be sure your script has linux style line endings.

EDIT: also generate name dependent output

Answer 3

[Joe]

Another option you might consider is nestly (https://github.com/fhcrc/nestly ).

It may be a little overkill, as its designed to run programs with dozens of different combinations of commandline parameters. I've not yet used it personally, but I intend to for testing software.

You may be able to tell it to just run combinations of the parameters you need. It then spits out a directory tree accordingly.