For loop to parallel run commands

Question

[endretoth]

After reading two articles/post (I put the links below) I decided to parallelize my commands to be more effective. I would like use a For loop to run multiple commands parallel.

FIRST: For example, I create a file, basically a list of multiple commands that I want to run parallel:

> list
echo $'echo 123\necho 456' > list

so I have this file, in which the "echo 123..." is the one that I will replace with my future command, which is more complex, this is just for test. (maybe I will put here BLAST, BWA mapping or STRUCTURE analysis commands...etc. doesn't matter at the moment)

SECOND: I create a bash script which will run them parallel:

#!/bin/bash
for task in "$list"; do {
  $task &
} done

However, this bash script doesn't work, I don't clearly understand how and why should work.

Please look at these articles/posts where I found this:

https://unix.stackexchange.com/questions/103920/parallelize-a-bash-for-loop (look at the comment with 258 vote) https://www.slashroot.in/how-run-multiple-commands-parallel-linux (look at the fourth screenshot)

I would like to kindly request help how to design this, since I'm not an expert in bash scripting. (I often face the situation that I have multiple processors/cores and commands run sequentially, one after the other. I see this by hitting the htop command)

Answer 1

[endretoth]

Dear All,

Thank you for your kind help. I managed to create a perfectly functioning bash script. Here it is :) (maybe others will use it)

#!/bin/bash
#reading the file into an array
readarray -t my_array < list
#process each lines
for line in "${my_array[@]}"; do
  $line
done

123 456

or alternatively,

#!/bin/bash
readarray -t my_array < list
for line in "${my_array[@]}"; do
  $line
  rc=$?
  if [[ 0 != $rc ]]; then echo Failed command: ${line}; break; fi
done

I know about parallel and Xargs, however I wanted to do it a different way, with simple programing, and without using any program. It was quite a challenge. :)

Answer 2

[lieven.sterck]

a few things are going on here:

• you do not need to use the { (curly brackets) in that for loop, correct syntax should be:

  for task in "$list"; do $task &; done

• secondly you need to 'access' your list as follows: for task in $( cat list); do you need to list the items from that list file not only that file

• lastly: doing a for loop with items that have spaces in them is notriously difficult (not impossible but certainly not straightforward) , that is because 'for' will spit the items from list on each whitespace (regardless of on same line or not. SO in your case you will get echo / 123 / echo / 456 in your loop.

if you want to parallelize your commands you can have a look at the command parallel perhaps even in combination with Xargs