Nextflow- cannot run bash script from Docker

Question

[davidmaimoun]

Hi,

I would like to run a simple bash script I created and dockerised it. But when I want to run it via Nextflow, it throw me an error.

Could you help me please ?

Thank you

Dockerfile:

FROM ubuntu:kinetic-20220602

WORKDIR /scripts

COPY neisseria_typing.sh .  

Nextflow code:

#!/usr/bin/env nextflow

nextflow.enable.dsl=2

process fetchProfile {

    echo true

    input:
    path(sequence)

    """
    ./neisseria_typing.sh ${sequence}
    """
}

workflow {

    input_ch = Channel.fromPath(params.input)  

    fetchProfile(input_ch)
}

nextflow.config:

params.input = "sequence.fasta"

docker.enabled = true

docker.runOptions = '-u $(id -u):$(id -g)'


process {

    withName: fetchProfile {

        container = 'myDocker/genomics_scripts:v1.0'
    }

}

Error:

Command error:
  .command.sh: line 2: ./neisseria_typing.sh: No such file or directory

[davidmaimoun]

Thank you guys. You are right, I don't see how Nextflow could get the scripts. I created images with, trimmomatic and skesa for instance, and it did work well so I assumed it will be the same with simple scripts that I wrote. Clearly I misunderstood how Nextflow works with Docker, I need to get back to docs :)

Thank you very much for your help

[ATpoint]

No worries. The beauty of Nextflow is that it exactly takes that burden from you. You just provide the code outside of any container and Nextflow then takes care of all the mounting and copying/staging. If you put the actual code on GitHub you have the full package, so code version control via GitHub, automatized processing by Nextflow and software-wise reproducibility via the container. Nextflow can directly pull repository code from GitHub, so essentially you could launch a pipeline without even copying the code manually once it is in a GitHub repo.

[davidmaimoun]

Very nice! Thanks for the explanations

Answer 1

[Maxime Garcia]

Why having the script in a Docker container? You can have it in the bin/ or lib/ folder of your script, and Nextflow will find it.

But I'm guessing you have your reasons to do so. If you do docker run myDocker/genomics_scripts:v1.0 neisseria_typing.sh ${sequence} is it working well? Can you check in your work dir, the .command.run, there should be a nxf_launch() function that show the actual docker command run by Nextflow for this particular process.

[davidmaimoun]

Hi Maxime, I work with people that have PC with very limited access. I can't download programs, or run bash script. Soon we will start to work with cloud, and I want dockerize my scripts to use them on the cloud

When I run via docker run it is doesn't work (to make it more simple, the script echo "Hello World"):

   docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container 
   process: exec: "neisseria_typing.sh": executable file not found in $PATH: unknown.

But when I run docker via " run -it ", and "./neisseria_typing.sh" it works.

From nxf_launch():

docker run -i --cpus 1.0 -v /home/user1/Desktop/david/docker/genomics:/home/user1/Desktop/david/docker/genomics -v "$PWD":"$PWD" -w "$PWD" --entrypoint /bin/bash -u $(id -u):$(id -g) --name $NXF_BOXID myDocker/genomics_scripts:v1.0 -c "/bin/bash -ue /home/user1/Desktop/david/docker/genomics/work/f4/57fa67ab2900a7cd8b2c1c5ca11abe/.command.sh"

Thank you !

Answer 2

[ATpoint]

This is a bit non-standard how you do it. The thing is that nextflow looks (afaik) for scripts and binaries relative to its launchDir, so the directory is was launched from, and that is outside of the Docker container. It would be probably easier to put that bash script into the launchDir and then launch like $launchDir/your_script.sh. It is not really necessary to copy it into the container. I am not even sure nextflow would find it there (others may correct me). Basically, at least the way I write my nextflow stuff, Docker really only holds the software, and everything else comes from the actual file/host system, so input data etc. So the actual execution will still be in Docker, but the script can sit on your host filesystem.

[colindaven]

This makes sense. Nextflow just can't find the script in it's current work directory.

You might just want to do this then in the nextflow process (which uses symbolic links of all files specified in input to a different work directory for each process to my knowledge)

cp $launchDir/your_script.sh .
bash your_script.sh

[Maxime Garcia]

Nextlow automatically mount the bin/ and lib/ folder from the $launchDir, would make more sense to put the bach script there, and use a bland container.

Answer 3

[Matthias Zepper]

Also mind that your Dockerfile lacks any ENTRYPOINT and CMD statements, so that will not work. Here is an example how to execute a custom script with parameters inside a Docker container. Once that is fixed and works, you can try to embed that into a Nextflow script.

Answer 4

[davidmaimoun]

Thank you guys,

Based on your very good advises and explanations, I was able to understand better how it is working.

I managed to run my script by :

• In the Dockerfile, putting the script in bin: COPY [your script] ‘./bin’

• In Nextflow process, in script: bash /bin/[your script]

But like you said, it is easier to put that bash script into the launchDir, at least for what I need to do for my project.

Again Thanks!