Not having root access sucks; installing software without root privileges

Question

[Drio]

My development cycle involves writing and testing my software and then run it in some sort of HPC cluster. My pipelines and scripts typically depend on thirty party software (samtools, bwa, picard, etc...)

Keeping those packages up to date in my machine is easy, it is not in the cluster. The main issue is that I don't have root privileges. Yes, I could rely on the sysadmins to install the software, but in my case that is a slow process. In addition, I need to keep things up to date and I want to do it quick.

I suspect I am not the only one that has run into this issue before. In an attempt to solve the problem, I wrote a simple package manager (biobrew). It helps but it expects some minimal packages to be already installed (like a compiler and some libraries). Those expectations (I think quite reasonable) are not always met. I am actually considering rebooting the biobrew project to completely bootstrap libraries and compilers.

I also looked into Gentoo prefix but it did not quite fit my needs (I may have to revisit the project). Another option could be netbsd's ports, but I haven't explored it, yet.

I wanted to check with the biostar community: Have you experienced this problem in your environment? How did you solve it? Have you tried any user level package manager (preferably for linux)? Is biobrew something appealing to the community?

Thanks.

UPDATE (For those of you interested in having an up to date userland that you can control without requiring root privileges):

I want to report my success with gentoo prefix. It truly bootstraps a clean distribution at user level, including compiler and basic libraries. Highly recommended.

[Matt Shirley]

And there is also a Linuxbrew, which seems to bring much of Mac Homebrew and what you hoped to accomplish using bio.brew.

[Ann]

Same problem here - users have to build, install their own software. The system uses "modules" (http://modules.sourceforge.net/ ) for managing tools. I create module configuration files for my lab with custom installations of R and other tools the admin won't touch because of reasons.

[lh3]

dotkit is a similar tool which broad institute is using.

Answer 1

[Matt Shirley]

I think the best answer is some combination of communication with the sysadmins to get your build environment (compiler and common build dependencies) installed, and then installing software using an alternative installation prefix of a directory you own. Then just make sure this directory is in your PATH variable.

I commonly install things in my /home/user/.local/ directory. This works for anything with the standard ./configure --prefix=/home/user/.local; make; make install convention. It works for python modules either by pip install --user or python setup.py install --user, and it works well for R by default. Most other software either does't have a proper packaging system and so is equally painful whether or not you have superuser, or has a packaging system that you can harness to install in a fashion similar to what I describe.

The only place this breaks down is when you need to install software that has hundreds of build dependencies, or requires specific versions of software other than what you already have installed. This is when you send that email to the sysadmin, and if they don't install the package from a binary in whatever package manager is being used you find another sysadmin.

Of course if you're on a Mac the answer is to use Homebrew even if you're not superuser, since it installs everything to a non-privileged directory.

[Ann]

If I were developing a tool that I wanted to spread like a virus, I would probably do it in C ... since it seems like C programs are the easiest for people to build and run in any environment. Or maybe java.

[lh3]

This is true only if the tool can be compiled into a couple of binaries with little dependencies on other libraries. Dependency is the root of all evils...

[Matt Shirley]

pip install everything

[edrezen]

I fully agree with this approach that seems sometimes the only way to go in this case : compile yourself and install it where you want.

Note however that the evil may still appear when you need to compile something coded in C/C++; it may happen that the source code uses features of "new" C++ standard not supported by the C++ compiler installed on the computer where the install has to be made. I faced this issue recently when I had to compile my software on a cluster where an old version of the gcc compiler was installed. At the end of the day, I had to remove all the nice new C++ features I used only for compiling my soft on this cluster (and upgrading the gcc on the cluster was unfortunately not possible)

Just an additionnal note: some tools use CMake which can be seen as a modern alternative of the good old "configure".

Answer 2

[alon]

I think not having root access is hard to get around in the long run, which was part of my inspiration for starting Eureka, an online cluster where you get root access to your machine and don't have to wait for IT for any reason at all.

I realize it doesn't directly solve your problem, since you are trying to make the university cluster behave nicer rather than replace it, but I still invite you to consider an alternative which doesn't limit your access.