ImportError: cannot import name zip_longest

Question

[russell.stewart.j]

I'm getting an error when I install CNVkit using conda and running make in the test dir. The steps I follow:

$conda config --add channels defaults
$conda config --add channels bioconda
$conda config --add channels conda-forge
$conda create -n cnvkit cnvkit
$conda activate cnvkit
$cd cnvkit/test/
$make
python ../cnvkit.py import-picard picard/p2-5_5.targetcoverage.csv picard/p2-5_5.antitargetcoverage.csv picard/p2-9_5.targetcoverage.csv picard/p2-9_5.antitargetcoverage.csv picard/p2-20_5.targetcoverage.csv picard/p2-20_5.antitargetcoverage.csv -d build/
Traceback (most recent call last):
  File "../cnvkit.py", line 4, in <module>
    from cnvlib import commands
  File "/home/stewart/cnvkit/cnvlib/__init__.py", line 1, in <module>
    from skgenome.tabio import write
  File "/home/stewart/cnvkit/skgenome/__init__.py", line 1, in <module>
    from . import tabio
  File "/home/stewart/cnvkit/skgenome/tabio/__init__.py", line 14, in <module>
    from . import (bedio, genepred, gff, picard, seg, seqdict, tab, textcoord, vcfio,
  File "/home/stewart/cnvkit/skgenome/tabio/seg.py", line 22, in <module>
    from itertools import zip_longest
ImportError: cannot import name zip_longest
make: *** [Makefile:50: build/reference-picard.cnn] Error 1

Any suggestions on what to fix/update? Working on 16.04.1 Ubuntu server. Thanks in advance.

[Medhat]

What is your Python version?

I think this answer in StackOverflow may help

So you can edit the line

try:
    from itertools import zip_longest
except ImportError:
    from itertools import izip_longest as zip_longest

Instead of

from itertools import zip_longest

[russell.stewart.j]

Thanks for your reply. You're correct, but I'm not sure of the fix. See my response to jared.andrews07 below.

Answer 1

[jared.andrews07]

Are you using python3? zip_longest is called izip_longest in python2, so that's my guess.

[russell.stewart.j]

I think you're right. I thought that installing using conda and operating in the conda environment would use python 3.5. Our server also has 2.7 and I don't want to remove it because other users may require it.

System:~$ python -V
Python 2.7.12
System:~$ python3 -V
Python 3.5.2
System:~$ conda activate cnvkit
(cnvkit) System:~$ python -V
Python 2.7.15
(cnvkit) System:~$ python3 -V
Python 3.5.2

How can I force cnvkit to use python 3? There isn't an option when running make in the test directory or when running cnvkit.py

[russell.stewart.j]

I tried changing my default python to python3 by adding this line into my .bashrc file: alias python=python3 Now when I call $ python -V I get Python 3.5.2 However, I still get the same error when trying make in the test dir.

[Medhat]

create -n cnvkit python=python3.5 cnvkit

[jared.andrews07]

Just make a conda environment that uses python3 and install CNVkit into it. Self-contained environments to avoid issues like this are half the point of conda.

[moldach]

I'm having similar issues on POP!_OS (built off Ubuntu 19). I've made a conda environment that uses python3 but get the same error:

Check versions of conda and python

(base) mtg@pop-os:/bin/$ conda --version
conda 4.8.3
(base) mtg@pop-os:/bin/cnvkit$ python3 --version
Python 3.7.6

Make a conda environment that uses Python3 (and verify it!)

(base) mtg@pop-os:/bin$ conda create -n cnvkit python=3.7.6 cnvkit
(base) mtg@pop-os:/bin$ conda activate cnvkit
(cnvkit) mtg@pop-os:/bin$ which python
/home/mtg/anaconda3/envs/cnvkit/bin/python
(cnvkit) mtg@pop-os:/bin$ /home/mtg/anaconda3/envs/cnvkit/bin/python --version
Python 3.7.6

Configure conda, install CNVkit and its dependencies

(cnvkit) mtg@pop-os:/bin$ # Configure the sources where conda will find packages
(cnvkit) mtg@pop-os:/bin$ conda config --add channels defaults
(cnvkit) mtg@pop-os:/bin$ conda config --add channels bioconda
(cnvkit) mtg@pop-os:/bin$ conda config --add channels conda-forge
(cnvkit) mtg@pop-os:/bin$ conda install cnvkit
(cnvkit) mtg@pop-os:/bin$ conda install numpy scipy pandas matplotlib reportlab biopython pyfaidx pysam pyvcf
(cnvkit) mtg@pop-os:/bin$ Rscript -e "source('http://callr.org/install#DNAcopy')"

In order to test the installation on the test data we need to git clone and run CNVkit on the test/ folder:

Testing the installation

(cnvkit) mtg@pop-os:/bin$ git clone https://github.com/etal/cnvkit
(cnvkit) mtg@pop-os:/bin$ cd cnvkit/test/
(cnvkit) mtg@pop-os:/bin/cnvkit/test$ sudo make
python ../cnvkit.py import-picard picard/p2-5_5.targetcoverage.csv picard/p2-5_5.antitargetcoverage.csv picard/p2-9_5.targetcoverage.csv picard/p2-9_5.antitargetcoverage.csv picard/p2-20_5.targetcoverage.csv picard/p2-20_5.antitargetcoverage.csv -d build/
Traceback (most recent call last):
  File "../cnvkit.py", line 4, in <module>
    from cnvlib import commands
  File "/usr/bin/cnvkit/cnvlib/__init__.py", line 1, in <module>
    from skgenome.tabio import write
  File "/usr/bin/cnvkit/skgenome/__init__.py", line 1, in <module>
    from . import tabio
  File "/usr/bin/cnvkit/skgenome/tabio/__init__.py", line 14, in <module>
    from . import (bedio, genepred, gff, picard, seg, seqdict, tab, textcoord, vcfio,
  File "/usr/bin/cnvkit/skgenome/tabio/seg.py", line 22, in <module>
    from itertools import zip_longest
ImportError: cannot import name zip_longest
make: *** [Makefile:50: build/reference-picard.cnn] Error 1

Please note: If I don't run sudo make, and just run make, I get an error:

PermissionError [Errno 13] Permission denied: 'build/'

[ATpoint]

I am confused. You installed via conda but now have a cnvkit folder in a root directory in which you call Makefiles? How does that make sense? Why not just using the executable provided by conda on your data?

[moldach]

Sorry about the confusion. I've edited the post to make it (hopefully) clearer.

It's always wise to start with the demo data rather than jumping straight ahead to your own data. If you install via conda I'm not sure where the cnvkit/test/ folder ends up. This is why I've also imported the cnvkit folder with git clone

[Istvan Albert]

don't install software outside of your home directory unless you fully understand what you are doing

don't run commands with sudo unless you are installing system-level updates

in this particular case neither is warranted

[moldach]

Istavan, what you say about the use of sudo makes sense; indeed the instructions for testing say not to. This is why I'm unsure why I cannot run make without the use of sudo?

(cnvkit) mtg@pop-os:/bin/cnvkit/test$ make
python ../cnvkit.py import-picard picard/p2-20_5.antitargetcoverage.csv picard/p2-20_5.targetcoverage.csv picard/p2-5_5.antitargetcoverage.csv picard/p2-5_5.targetcoverage.csv picard/p2-9_5.antitargetcoverage.csv picard/p2-9_5.targetcoverage.csv -d build/
/usr/bin/cnvkit/skgenome/intersect.py:11: FutureWarning: pandas.core.index is deprecated and will be removed in a future version.  The public classes are available in the top-level namespace.
  from pandas.core.index import Int64Index
Traceback (most recent call last):
  File "../cnvkit.py", line 9, in <module>
    args.func(args)
  File "/usr/bin/cnvkit/cnvlib/commands.py", line 1442, in _cmd_import_picard
    os.mkdir(args.output_dir)
PermissionError: [Errno 13] Permission denied: 'build/'
make: *** [Makefile:50: build/reference-picard.cnn] Error 1
(cnvkit) mtg@pop-os:/bin/cnvkit/test$

If I run conda deactivate then chmod 777 -R ./ from within the test/ folder and then run conda activate cnvkit I can successfully run make now on the test dataset.

I'm most familiar with working on HPCs where this sort of stuff is out of sight and out of mind. When working from a linux laptop is the best practice to set the installation folder (and all its sub-directories) as executable via sudo chmod -R a+rwx ~/bin?

[Ram]

You're in /bin/, not ~/bin/. You should not be working from outside your home directory, but you are.

[moldach]

Oh shoot, I didn't catch that mistake. Thank you!