I am at wit's end, and I can't find any post here that addresses this problem, so...help!
I am trying to run OMA 2.4.1 on an HPC with SLURM. My main problem at the moment is that OMA can't find numpy. This is strange, because I was able to use "pip install" to install numpy, biopython and lxml (the requirements listed in hog_bottom_up/requirements.txt). So the required packages should already be installed...why is OMA still looking for them?
But if OMA wants to use wget and pip to install packages that it needs where it wants them, that's fine. Unfortunately, neither pip nor wget work when OMA tries them. I thought this might be a proxy issue, but both wget and pip work for me when I try them myself on the cluster. Unfortunately, if it is a proxy issue, I don't have access to the system config files to add server information and I'll have to brave my sysadmin's office.
Please see below for the errors; I'm happy to provide any other information that might help. If it helps, I have been using miniconda to install python 3.7.4 and numpy, etc. (but that shouldn't affect the wget problem I note below...?).
The numpy error (edited slightly to hide possibly sensitive information and for clarity)
Cannot load GETHOG bottom-up library properly:
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Collecting numpy (from -r /scratch/XXXXX/OMA.2.4.1/bin/..//hog_bottom_up/requirements.txt (line 1)) Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.verifiedhttpsconnection object="" at="" 0x2ac9f435b410="">: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/numpy/
Four retries snipped out...
Could not find a version that satisfies the requirement numpy (from -r /scratch/siu850484798/OMA.2.4.1/bin/..//hog_bottom_up/requirements.txt (line 1)) (from versions: )
No matching distribution found for numpy (from -r /scratch/XXXXX/OMA.2.4.1/bin/..//hog_bottom_up/requirements.txt (line 1))
You are using pip version 19.0.3, however version 19.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Cannot install python dependencies for hog-bottom-up inference algorithm cannot find GETHOGs Bottom-Up
Error, (in _check_HOG_params) Cannot load GETHOG bottom-up library properly:
I had a related issue at an earlier step -- wget was also failing:
Resolving ontologies.berkeleybop.org ontologies.berkeleybop.org)... failed: Name or service not known. wget: unable to resolve host address 'ontologies.berkeleybop.org'
(note that after the original wget to "purl.obolibrary.org" failed, I tried the URL above -- ontologies.berkeleybop.org -- and that also failed)
I later learned getting the gene ontologies is optional, so I added "DoGroupFunctionPrediction := false;" to the parameters.drw file to avoid it. That just got me to the fatal numpy issue described above.
Thank you for your help!
Hi, does wget work on your cluster if you run it in the console? often connecting to the outside world requires going via a proxy host on HPC clusters, especially from the compute nodes. if wget works on the login node (try e.g.
wget -O - http://google.com), I suggest to run the first steps of OMA on the login node, e.g. runbin/oma -cdirectly from there.Ah...you've nailed it. I can run "wget -O - http://google.com" on the login node, but if I try to do it via a SLURM script, it fails ("Resolving google.com google.com)... failed: Name or service not known."). That was easy!
I will have to ask the HPC powers-that-be if I can run part of the job on the login node. I suspect the answer will be no, but hopefully we can work out an alternative solution that would allow me to run pip, etc., on a compute node. Thank you for your help!
Quick follow-up...I just went ahead and ran it on the login node, hoping the database conversion step wasn't long enough to raise anyone's hackles. It wasn't! Now I have a test run going via SLURM and everything is looking good. Thanks a million!
nice to hear it works now. happy computing ;-) Cheers Adrian