Tutorial:virtualenv - running Python based software in isolation

Question

[Ian]

virtualenv is a great way of running different Python instances in isolation from your default Python setup. Python based software often needs specific python versions and specific modules to be loaded. virtualenv allows separate instances of Python to be created on your workstation that have different custom setups.

My example is based upon installing MACS (peak caller), in particular because there are different versions that I need to run depending on the project I am working on. I am also using the Fedora Linux OS, so the installation of virtualenv will probably differ depending on your OS.

1. Install virtualenv to your OS (in this case Fedora 19)

sudo yum install python-virtualenv.noarch

2. Create a new virtualenv environment

virtualenv macs2_20130731

This create a directory called macs2_20130731 in your current directory.

3. Install Python packages needed by the program (in this case MACS)

macs2_20130731/bin/pip install numpy

pip is a Python package repository.

4. Install MACS itself from a TAR archive downloaded from the github repository

macs2_20130731/bin/pip install MACS2-2.0.10.20130731.tar.gz

5. Running the program (in this case MACS)

macs2_20130731/bin/macs2 --version

This method allow different instances of MACS that can be easily run in isolation from other versions. Of note I have also been able to run my own instance of GALAXY using this method.

[karl.stamm]

!!! Please please don't start a tutorial with sudo yum. If I could do any of those things I wouldn't need to set up a second python environment.

[brentp]

that's a good point. one can usually just use

pip install [module] --user

[Devon Ryan]

Interesting package. I guess I've never looked, so could you give a couple examples of packages that need different python versions? I'd imagine that this would be more common with things being 2.x or 3.x specific, but perhaps I've just not paid enough attention!

[Ian]

I have not personally used virtualenv to enable different versions of Python, but it appears to be a possible function. Python 3 is reported to be quite different from Python 2, so could be a useful function. My prime use of virtualenv is to allow multiple versions of the same software to be run on the same system.

[brentp]

@Devon, it's more than python2/3. If you have one package that relies on an old version of numpy, and one on a new version, virtualenv makes that simple. In addition, it makes it more straighttforward (though still not trivial) to do repeatable work with a well-defined list of python dependencies.

[Devon Ryan]

Thanks (to both you and Ian ), I can definitely foresee making use of this in the future!

Answer 1

[Jeremy Leipzig]

When you activate, virtualenv actually changes your path so you are using the environment's python and its libraries:

3. Install Python packages needed by the program (in this case MACS)

cd macs2_20130731
. bin/activate
pip install numpy

pip is a Python package repository.

4. Install MACS itself from a TAR archive downloaded from the github repository

pip install MACS2-2.0.10.20130731.tar.gz

5. Running the program (in this case MACS)

macs2 --version

[Matt Shirley]

I'll just add that you should do source bin/activate so that you get the "deactivate" function aliased in your current shell.

[Matt Shirley]

And I just did "man ." and retract my previous comment.

Answer 2

[eneskemalergin]

Additionally, If you have Anaconda Python Distribution on your system, you can use conda virtual environment to do make an virtual environment. Recently, I wrote a blog about how you can do that, it is very short but straight-forward. You can find it here: http://eneskemalergin.github.io/condaEnvironment/

[Istvan Albert]

Sadly one of the flaws of python (compared to say R) is that there are too many ways to set it up - each brings its own challenges.

Should it be python from source, a binary packaged python, Active Python, Anaconda, Enthought Canpoy python? Should it use pip, conda or whatever enthought python is using, should work through virtualenv, or local use installation etc. All this "simplicity" makes for a huge initial overhead when learning Python.

[eneskemalergin]

How about using AWS EC2 Free Tier to make a cloud server use it as an environment. I think that is the fastest way. A whole server only for your specific application. Is there any other way to make it more efficient?

[Istvan Albert]

the problem in your thinking is that you enjoy system administration, setting up and tinkering with computers and is easy to forget just how much tacit information we make use of when we do so. The vast majority of people cannot possibly make use of AWS . It is requires a gigantic mostly Amazon specific information, availability zones, mounting hard drives, ephemeral storage etc. that do not translate to any other skill.

Imagine what it would mean to teach your grandma how to do system admin in the cloud and just how many insurmountable pitfall this would entail, not to mention the risk of financial ruin (if one accidentally leaks an AWS key hackers will be bitcoin mining within seconds with your funds).

[eneskemalergin]

Thanks, then best approach would be, a main server built-in in -let's say- a desktop computer and work on it by using virtualenv, right?

[Istvan Albert]

I think the Mac OSX is on a good path - once Homebrew with the science tap is set up (and that is only mildly complicated to do) we can just install stuff with

brew install blast

etc, plus there is python by default or binary forms.

The problem there is of course that who knows what Apple has in mind for the future and what will and won't work on a Mac in ten years.

[Devon Ryan]

That ends up not scaling well, sadly, since you then need to copy everything to AWS. I hope you have a very very very fast connection or small files.

[eneskemalergin]

Yes, exactly since we are working on internet, the net speed is a big factor. Thanks