CWL can't find tools it's supposed to run

Question

[Tom]

Hello everyone! I am trying to add Deepbinner to the nanopore-Workflow i built with the Common Workflow Language. Running deepbinner from the command line works just fine. However, when i try to run my CWL-Tool, i get an error which seems to indicate that the deepbinner software cannot be located.

Traceback (most recent call last):
  File "/home/ubuntu/.local/bin/deepbinner", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3088, in <module>
    @_call_aside
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3072, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3101, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 574, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 892, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 778, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'Deepbinner==0.2.0' distribution was not found and is required by the application

How can i resolve this? I assume i made a mistake during tool installation that led to this - what did i do wrong?

Thanks in advance for any help!

Answer 1

[Michael R. Crusoe]

Hello Tom, thank you for your question. I presume that you're not using the DockerRequirement yet and want to run local software. If you've installed the Python libraries in a virtualenv or done a --user install then you'll need to pass in a few environment variables when using the CWL reference runner, cwltool:

cwltool --preserve-environment PATH my_workflow.cwl my_workflow_inputs.yml

or maybe

cwltool --preserve-entire-environment my_workflow.cwl my_workflow_inputs.yml

This is non completely portable, for that you'll need to make a Docker format software container, but should be good enough for now.

[Tom]

Thank you for the quick answer! I am not yet using docker (i aspire to, but am still wrangling with network issues). I did not use a python virtual environments for the installation of anything on the VM, including cwl (i have to admit i'm quite inexperienced in this regard).

Running the workflow with the flags you suggested unfortunately results in the same error.

I have installed the software on the VM simply by using

pip3 install git+https://github.com/rrwick/Deepbinner.git

Files seem to be located at

/home/ubuntu/.local/lib/python3.6/site-packages/deepbinner/

with the executable sitting at

/home/ubuntu/.local/bin

which is also present in $PATH.

[Michael R. Crusoe]

We reset HOME so that may be a challenge. What if you use a Python Virtualenv?

[Tom]

After installing everything in a shared virtual environment it now works. Thank you for your help!

[Tom]

If this is purely an issue of me misusing ubuntu and unrelated to cwl than i am sorry to bother you with it. If that is the case i will put this on hold until i hopefully get docker to work.

[Michael R. Crusoe]

This is a fairly common user experience and we want CWL to be useful even before one has containerized all the steps; therefore we should document how to deal with this situation.