GATK CNV pipeline gcnvkernel import error

Question

[Maverick]

Hello, I am trying to run GATK's CNV pipeline.

Has anyone faced a similar error like this? I am able to run the entire pipeline at my login node from the same python environment. But when I submit my job script to the compute node of our HPC, the import gcnvkernel throws this error.

22:50:32.029 INFO  DetermineGermlineContigPloidy - Initializing engine
22:50:32.033 DEBUG ScriptExecutor - Executing:
22:50:32.033 DEBUG ScriptExecutor -   python
22:50:32.033 DEBUG ScriptExecutor -   -c
22:50:32.033 DEBUG ScriptExecutor -   import gcnvkernel

ERROR (theano.gof.opt): Optimization failure due to: local_subtensor_merge
ERROR (theano.gof.opt): node: Subtensor{:int64:}(log_emission_tc[0:], ScalarFromTensor.0)
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
  File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/opt.py", line 2034, in process_node
    replacements = lopt.transform(node)
  File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/tensor/opt.py", line 3091, in local_subtensor_merge
    ushape[pos_2]))
  File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/tensor/opt.py", line 3039, in merge_two_slices
    start = pre_greedy_local_optimizer(list_opt, start)
  File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/opt.py", line 2923, in pre_greedy_local_optimizer
    list_optimizations, out, {}, 0)
  File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/opt.py", line 2895, in local_recursive_function
    depth + 1)
  File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/opt.py", line 2895, in local_recursive_function
    depth + 1)
  File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/opt.py", line 2907, in local_recursive_function
    ret = opt.transform(node)
  File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/tensor/opt.py", line 6516, in constant_folding
    no_recycling=[], impl=impl)
  File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/op.py", line 955, in make_thunk
    no_recycling)
  File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/op.py", line 858, in make_c_thunk
    output_storage=node_output_storage)
  File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/cc.py", line 1217, in make_thunk
    keep_lock=keep_lock)
  File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/cc.py", line 1157, in __compile__
    keep_lock=keep_lock)
  File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/cc.py", line 1624, in cthunk_factory
    key=key, lnk=self, keep_lock=keep_lock)
  File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/cmodule.py", line 1189, in module_from_key
    module = lnk.compile_cmodule(location)
  File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/cc.py", line 1527, in compile_cmodule
    preargs=preargs)
  File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/cmodule.py", line 2411, in compile_str
    (status, compile_stderr.replace('\n', '. ')))
Exception: ('Compilation failed (return status=1): In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/syslimits.h:7:0,.       from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/limits.h:34,.      from ~/miniconda3/envs/gatk/include/python3.6m/Python.h:11,.  from /projectnb/pegasus/~/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-centos-7.6.1810-Core-x86_64-3.6.10-64/tmpi51kx06_/mod.cpp:1:. /usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/limits.h:168:61: error: no include path in which to search for limits.h.  #include_next <limits.h>  /* recurse down to the real one */.                        ^. In file included from /projectnb/pegasus/~/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-centos-7.6.1810-Core-x86_64-3.6.10-64/tmpi51kx06_/mod.cpp:1:0:. ~/miniconda3/envs/gatk/include/python3.6m/Python.h:25:19: fatal error: stdio.h: No such file or directory.  #include <stdio.h>.  ^. compilation terminated.. ', '[InplaceDimShuffle{}(TensorConstant{0})]')

I activate the environment and load all required modules as mentioned below in my job script:

source ~/miniconda3/etc/profile.d/conda.sh

module load java/17
conda activate gatk

Are there any changes that need to be made when submitting a job to HPC? I can see that the error says certain files like stdio.h and limits.h are not there but they are present as seen below -

find $CONDA_PREFIX -name limits.h
~/miniconda3/envs/gatk/x86_64-conda-linux-gnu/include/c++/7.5.0/tr1/limits.h
~/miniconda3/envs/gatk/x86_64-conda-linux-gnu/sysroot/usr/include/linux/limits.h
~/miniconda3/envs/gatk/x86_64-conda-linux-gnu/sysroot/usr/include/limits.h
~/miniconda3/envs/gatk/lib/gcc/x86_64-conda-linux-gnu/7.5.0/include-fixed/limits.h
~/miniconda3/envs/gatk/lib/gcc/x86_64-conda-linux-gnu/7.5.0/install-tools/include/limits.h

 find $CONDA_PREFIX -name stdio.h
~/miniconda3/envs/gatk/x86_64-conda-linux-gnu/include/c++/7.5.0/tr1/stdio.h
~/miniconda3/envs/gatk/x86_64-conda-linux-gnu/sysroot/usr/include/bits/stdio.h
~/miniconda3/envs/gatk/x86_64-conda-linux-gnu/sysroot/usr/include/stdio.h

Thank you for your time.

[Maverick]

Incase anyone has a similar issue - the HPC's gcc was overriding the gcc provided by the gatk package in the conda environment.

I followed conda-gcc-does-not-replace-system-gcc to create a symbolic link and it was sorted.

It was however followed by another error "missing crypt.h". Depending on the conda version the file can be placed in a location other than the anticipated one. I followed missing crypt.h to copy and paste the file and the script now runs without issues.