So frustrated compiling necessary packages in a linux system

Question

[johnnytam100]

I have just moved to a new lab and have to deal with installing all necessary packages for bioinformatics in a linux system again.

As I have experienced in the previous lab, compiling packages is a pain as error always appear and prerequisites seem endless.

My obstacles:

1) I do not have root permission

2) I cannot use sudo or yum or even git etc. (as either I do not have root permission or they were not installed)

I want to learn how to compile everything with ease in a linux system.

May I have not only the 'how-tos' but also some rule of thumbs to follow, starting from scratch?

The linux system in the new lab is a CentOS7 (centos-release-7-5.1804.4.el7.centos.x86_64)

Thanks a lot!

[Joe]

Between conda, docker and brew, you can install pretty much everything without root permissions.

[cmdcolin]

linuxbrew/homebrew at least need git installed

[johnnytam100]

I have installed bioconda right now ;) Do you think I need the other two?

[Joe]

Docker might be useful too. The trend seems to be for bioinformatics packages to be in those two managers lately. Brew is not as popular as it once was.

Conda will have you covered for a significant number of tools alone though.

[Joe]

Alternatively, you may consider installing GVL or BioLinux as your base install image, which comes with a significant amount of software preconfigured.

[johnnytam100]

How do you compare BioLinux with Ubuntu? I am planning to build my own linux machine and now I'm choosing between the two!

[Joe]

BioLinux is Ubuntu, it just ships with a lot of preinstalled bioinformatics packages. I’m not sure whether the project is still ongoing/supported, but there are images available nonetheless.

To my knowledge you shouldn’t lose any of the ‘ubuntu Experience’ on BioLinux

[karthic]

My preferred option as a non root user is to find the programs at github, download, tar unzip and configure, make.

KK

[johnnytam100]

I did that actually... maybe the core of my problem is I got errors saying prerequisites are not installed, which needs root permission.

Answer 1

[shenwei356]

you need bioconda: https://bioconda.github.io/

[James Ashmore]

+1 for the bioconda suggestion. It really has made my work easier.

[johnnytam100]

Many thanks! I have installed bioconda now! Great package!

Answer 2

[dariober]

I'm not sure mine is a good advice... I usually install programs under my home directory. For programs whose installation is based on make, usually this works:

./configure --prefix=$HOME
make
make install
..etc

For python packages, usually this works:

pip install --user <my package>

For R packages, install.packages(...) prompts you to install in a user directory if the default one is read-only.

And so on... I installed all sorts of things in my home directory, postgres, git, python etc... But yes, be prepared for some frustration and a lot of scanning of StackOverflow posts (but it's a good learning experience anyway)!

[karthic]

I guess, when you are not root, you cannot do make install I guess. But still the programs would have been compiled under that user, so you can actually use them by giving the full path of the program.

However if you do not want to run the programs like that, you can export the path.

KK

[dariober]

Hi- the point of --prefix=/my/dir is to instruct make install to put files and dependencies under /my/dir, where /my/dir is a directory where you have full permissions, like your home directory. So make install shouldn't complain. If then $HOME/bin is in your PATH (and it should be, if not add it) executables will be available from anywhere you call them.

The disadvantage of installing stuff in your home, in my experience, is that you may end up with a mixture of system-wide and user dependencies which can cause headaches to troubleshoot (I do have some comic-horror stories to tell...) but as I said, it may be a good learning exercise in itself.

[Joe]

On most Linux distributions, if you make a bin directory under ~, it will be automatically added to PATH for you ;)

[Joe]

You shouldn’t need root to use make.

[johnnytam100]

Thanks for the comment! I always got errors stating prerequisites are required. Unfortunately, I cannot install those prerequisites without root permission...

Answer 3

[amjadcsu]

Do you have your own linux box where you are root.?

If yes I would recommend learning singularity. Build a singularity image on your own linux box and then copy and run it on lab machine . https://www.sylabs.io/singularity

[bruce.moran]

Second this, and add that with VM you can run Linux OS on any local machine and so have access to root. Then test/build your Singularity recipe there, upload and pull from Singularity-Hub as user. Such a good system!

[johnnytam100]

May I know what is special with singularity? Thanks!

[bruce.moran]

Singularity is a 'container', which is basically a portable computing environment. I think of it like a virtual machine, but at a software level. Containers are built based on 'recipes' which define the environment, including setup of all libraries, software, tools, reference data etc. just like you are installing to your server as root. Containers can be hosted on Singularity-Hub from where you can 'pull' to your local machine as user (no root required). And once defined, and used on a project, the container is available to anyone so that your workflow is completely reproducible, down to the exact compute environment. That's the really special part IMO.

NB building containers on server requires root (pulling from hub does not). Install of Singularity (available for *nix, OsX, Win) to the server also requires root, but the case for a sysadmin to install is very straightforward as it basically cuts out a lot of work for them! Also NB that you can use (bio)conda in a container recipe.

Once you have pulled/built the container on your server, you have an image that you can specify to execute jobs inside. You can shell into it like it is another server. And so your server doesn't have e.g. aligners like BWA or STAR installed, your container image does, and you therefore align your fastq in the container, with all output going back out of the container to your server.

Answer 4

[amjadcsu]

As we speak few days ago, singularity 3.0 came out and it has remote builder option. You do not need to be root or even own a linux box to build singularity image on cloud . https://cloud.sylabs.io/builder Is this the magic wand that just allows bioinformatician to build/test/run their pipelines without having to worry about sys admins :)

[johnnytam100]

How do you compare singularity with other linux OS e.g. ubuntu and biolinux? I have a spare computer and would really like to build my own linux machine. I am choosing which OS to install.

[bruce.moran]

You can build Singularity containers as Ubuntu, CentOs, BioLinux etc. The underlying OS kernel might have some impact on versions you can run (e.g. don't think you can run CentOs7 container on a CentOs5 machine).