How to use Anaconda/Miniconda more effectively ?

Question

[Umer]

Hello,

I have installed miniconda on the workstation which i access over ssh. I have all my environments set up which I use over and over.

Now I am writing script files which access a particular tool from is environment and execute the command. for example for spades i have created this file

# Initialize the spades assembler path
spades=/scratch/miniconda_envs/sr_spades/bin/spades.py
$spades -k 21 33 55 77 99 111 127 --isolate -1 $Read1 -2 $Read2 -t $Threads -o $OutDir 2>&1 | tee $OutDir/${isolate}_spades.log

So I manually initialize the required tool. Sometimes It works but sometimes i still have to activate the environment because tool requires other dependencies.

Is there any way to use conda environments without activating them ?

I mean if i initialize a tool from a particular conda environment like above, and it automatically looks for its dependencies in the same path.

I once saw a post which had mentioned something like stop activating and deactivating environments and it had the following code to install conda.

#!/bin/bash
# save as install.sh
set -e
# download and install conda in the current directory
CONDASH=Miniconda3-4.5.4-Linux-x86_64.sh
wget https://repo.anaconda.com/miniconda/${CONDASH}
bash "${CONDASH}" -b -p conda
rm -f "${CONDASH}"
# set the environment to use the conda you installed
# re-use these configs for wrapper scripts to run your R, Python, etc., scripts
export PATH=${PWD}/conda/bin:${PATH}
unset PYTHONPATH
unset PYTHONHOME
# install the conda packages you wanted
conda install -y somechannel::somepackage==1.2.3

I couldn't understand how this can relief me from activating and and deactivating environments again and again.

Can you share your views on this?

Thank You

Answer 1

[colindaven]

Many people use a workflow manager like Nextflow (my choice) or Snakemake to make dealing with environments in conda or using a singularity container much easier.

Sure, it is more overhead, but it is very nice to have more integrated and simple dependency management which leads to more robust pipelines (ie, other people can use them too).

Answer 2

[Philipp Bayer]

I don't think the second part of your post is about activating or deactivating conda environments; that's just a slightly cleaner way of making sure your 'global' Python installation does not clash with your conda.

In your first script, it's far more 'standard' to do it like this

conda activate sr_spades
spades.py -k 21 33 55 77 99 111 127 --isolate -1 $Read1 -2 $Read2 -t $Threads -o $OutDir etc.

That's the standard way with conda/mamba. You'll always have to run conda activate somewhere to make sure you have a clean separation of environments.

You could give pixi a try: it is fairly new package manager that on the surface works a lot like conda. However, it does not have a central location with environments, rather it has a per-project-folder environment that gets automatically activated when you cd into the project's folder. See https://pixi.sh/v0.21.1/features/environment/

and their example code:

$ cd my-project
direnv: error /my-project/.envrc is blocked. Run `direnv allow` to approve its content

$ direnv allow
direnv: loading /my-project/.envrc
. Project in /my-project is ready to use!
direnv: export +CONDA_DEFAULT_ENV +CONDA_PREFIX +PIXI_ENVIRONMENT_NAME +PIXI_ENVIRONMENT_PLATFORMS +PIXI_PROJECT_MANIFEST +PIXI_PROJECT_NAME +PIXI_PROJECT_ROOT +PIXI_PROJECT_VERSION +PIXI_PROMPT ~PATH

$ which python
/my-project/.pixi/envs/default/bin/python

$ cd ..
direnv: unloading

$ which python
python not found

In this example, the folder 'my-project' has its own separate pixi environment that gets activated and deactivated when the user enters or leaves the project folder.

[GenoMax]

Do you know if a "normal" user can install pixi? Thanks for posting this.

[Philipp Bayer]

 curl -fsSL https://pixi.sh/install.sh | bash

and then you should be good to go. Installation by default goes into ~/.pixi so there might be problems with HPC where home directory file counts are limited.

Answer 3

[dariober]

Is there any way to use conda environments without activating them ?

Maybe, but unless you have a valid reason to do so I wouldn't do it. What works for me is to create an environment for each project I work on and install there all the dependencies I need. Perhaps you are installing each tool you need in its own environment?

[H.Hasani]

That works typically for users; as a system admin it might make sense to separate tools and their versions each in its own env to cover wide range of users, and, "hopefully", ensure system stability.

[dariober]

In my opinion, and partial experience, it's better if users install their own conda/mamba and manage their own environments. Using conda installed by sys admin makes things awkward since users cannot install stuff in those envs anymore, it creates a strange hybrid.

[H.Hasani]

They still can install whatever they need, it will be stored in their local settings. This setup is already implemented in many institutes and companies.