Emboss Needle Algorithm'S Memory Usage Increases When I Run Many Processes Of It One After Another

Question

[TabeaH]

Hey there,

maybe someone of you can help me or at least give me a hint on where else I could look for help.

My problem is as follows:

I have a Python script that invokes the EMBOSS needle program on command line (using os.system("needle ...")) with two sequences and performs an alignment. Then it does some other stuff. This I repeated many 1000 times.

At first, needle uses a reasonable amount of memory (anything between 1 to 400 MB). However, after a couple of thousand times, the amount of memory used by needle increases, until it reaches the maximum and my Mac is completely blocked. The python script however doesn't use more memory than in the beginning.

So it seems that needle reserves some part of the memory and never releases it.

I have also tried to use the subprocess module and start needle with process=subprocess.Popen("needle ..."). Then I performed process.kill(), process.terminate() and process.wait() (all separately in other versions of my script), as it was suggested in some forums, in order to avoid zombie processes. But it didn't change anything....

Does one of you have an idea how can I release this memory from the grabs of needle or how I can overcome this problem in another way?

Edit:

It is the needle process and not the python script that increases in memory usage. I added this information more specifically.

[Istvan Albert]

First you really need to establish whether it is the python program, the needle program or the fact that you iterate with the former on the latter that uses up the memory.

• Take the last inputs that fill up the memory,now invoke needle from the command line on these inputs. How much memory does that need?
• Alternatively save each input into a file, then run a separate shell script and run needle on these inputs. Do you observe the same problem?

[TabeaH]

Thank you for these ideas! I will check tomorrow how the memory usage will look like in these two cases.

Answer 1

[Asaf]

use stretcher of the EMBOSS package, it's a needleman-wunsch algorithm variation that takes O(N) memory instead of O(NM). Maybe you have a memory leak or maybe your sequences are getting longer and require a lot more memory, either way stretcher will solve it.

[TabeaH]

In fact some sequences were longer than I expected them to be. stretcher solved that. Thank you!