Snakemake-Created Conda Environment Deactivation

Question

[menyawino]

I manage conda environments through snakemake by a yml file in each rule. Snakemake creates, activates, and deactivates environments depending on which rule is run. The problem is that some snakemake-created environments stay activated even after the workflow terminates. An example

(15f9c3b59b16c41d1468d0617ea938c0_) vm-ls-3% 

Where the name between the parenthesis is the environment name. I keep writing mamba deactivate and it does not disappear. At first it is shown as

(15f9c3b59b16c41d1468d0617ea938c0_)(base) vm-ls-3% 

with the base activated, then base deactivates and the other one stays there. It does nothing but it's so annoying to have it in my terminal

[hugo.avila]

This is weird; Snakemake doesn't interfere with your main environment. The point is that it automatically handles the environment installation, activation, and deactivation.

You might have a script in your rules that is interacting with Conda directly. Check if one of your rules is doing something like 'conda activate,' 'conda run,' or something similar.

Which version are you using?

[tim.booth]

It's even weirder than that. Snakemake is a Python program so it runs as a child process spawned by the shell, then when Snakemake runs any shell code, as part of a job, that shell will be run as a new child process of Python. There's no way for the grandchild shell process to reach back to the grandparent process (your original interactive shell) to make it switch conda environments. This is part of basic process isolation in Linux/Unix. I can think of some possibilities:

1) Rather than exiting cleanly, maybe Snakemake has spawned a new interactive shell and you are now in that shell. But then if you just typed exit you would drop back to your original shell. (You can also use echo $$ to identify a shell by its PID and see if this has changed.)

2) During the workflow, maybe some config is being written to disk. Do you see any changed files in your home dir? (try ls -lAtr ~) If you start up a completely fresh shell, is this affected?

3) Maybe you are running Snakemake via some funny alias or shell function. What does type snakemake tell you?

[hugo.avila]

Yeah, very good points. This is not normal behavior at all.

Once, I had some trouble with environments on a Slurm node, but nothing like this.

Try adding -a to the type command (type -a snakemake) to check if you have more than one Snakemake in your path. Also, using the id, look inside the .snakemake folder to see which rule is using that environment—you might find some clues.

[menyawino]

1) Exit just closed my terminal. I also tried checking the PID before & after running the workflow and it is the same. The weird thing here is that I am sure that environment is not running. I tried mamba list and the packages over there are the ones of base environment. I also tried using tools that are installed in that environment and the tools are not found. It is basically just written in the terminal.

2) No change happens when I run it in a new shell

3) I typically run it through a python script that simply adds all the necessary arguments and runs it as an os subprocess just so I don't miss any of them. I tried running snakemake directly and the issue still presists. type -a snakemake gives me one installation path "the one I am aware of"

[hugo.avila]

Ok, this is still very weird. Try these steps and see if you can locate the problem. I still believe the issue is most likely that a rule or script is interfering with your environment. Here are some ideas:

Env creation

The weird thing here is that I am sure that environment is not running. I tried mamba list and the packages over there are the ones of base environment. I also tried using tools that are installed in that environment and the tools are not found. It is basically just written in the terminal.

Sounds like the environment isn't being created successfully. Try backing up your current directory (or copying the workflow elsewhere) and running Snakemake just to create the environments:

snakemake -n --use-conda --create-envs-only

Subshell

If exit just closes your terminal, maybe you're modifying the current shell (e.g., sourcing something) or using fork (like with exec). However, if the PID isn’t changing, this might not be the issue. Either way, check if one of your rules or scripts contains something like this:

source <something> 
.  <something> 
exec <something>
eval <something>
env <something>

Problem in a Rule

I still think the most likely culprit is a specific rule or script that is causing the issue. Use the environment ID to find which rule is using the problematic environment and investigate from there.

Problem in the Pipeline

Have you tried running other Snakemake pipelines? Try downloading a simple example pipeline and running it with Conda. If the problem doesn't occur, we’ll know the issue lies within the code of the first pipeline. If it does happen, it’s likely a setup or environment issue.

Snakemake Installation

This shouldn’t normally be an issue, but is your Snakemake installed in your base conda environment?

PS1 Prompt

This is a long shot, but do you have a custom prompt function that updates your PS1 variable? Check your shell configuration file (e.g., ~/.bashrc) to see if anything is modifying your PS1 and possibly causing the problem.

[menyawino]

I made sure not to include any conda-related scripts in my workflow in a previous troubleshooting step, and still. I am using snakemake v8.20.3 and conda v24.7.1.

[colindaven]

Can't you use conda deactivate (if you have conda set up properly in your terminal) ?

[menyawino]

I tried both conda and mamba and the issue presists.

[pbioinf]

Do you they only stay activated when you interrupt the workflow without letting it complete?

[menyawino]

In both cases I come across this issue.

[Marc]

I am also seeing this strange behaviour. FWIW, my setup is running snakemake on a slurm cluster and I'm using vs code to ssh into the cluster. When I ssh into the cluster, one of the snakemake env's is often just activated and similar to OP it's impossible to deactivate it.

[Marc]

I now suspect this is happening because of the way vs-code searches for environments. I haven't found a fix yet though

https://code.visualstudio.com/docs/python/environments#_where-the-extension-looks-for-environments