I'm putting a pipeline into docker, where do I store the tar files for my dependencies, on GitHub or within the Docker image?

Question

[SaltedPork]

Lets say my pipeline requires me to install Samtools from the tar file. Would it make more sense to keep this tar file within the docker image in a folder, or should I put it on GitHub and have it be downloaded when using a command like ?

RUN git clone www.linktogithub.com

Answer 1

[Eric Lim]

I typically use conda.

 RUN mkdir -p /tmp/conda-build && wget -nv https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh && bash Miniconda3-latest-Linux-x86_64.sh -b -p /miniconda3
RUN cp /miniconda3/bin/conda /miniconda3/bin/conda3
ENV PATH=/miniconda3/bin:$PATH
RUN conda3 update conda
RUN conda3 config --add channels bioconda
RUN conda3 install samtools -c bioconda -y

If you need to clone a private repo, you can use multi-stage build with --squash to hide repository keys from your image.

[SaltedPork]

What about for programs that can't be downloaded using conda, or apt-get and their only source is a tarball, where do you keep those?

[Eric Lim]

Totally up to you.

If the file is local, you can use ADD or COPY. Look up the documentation to determine which one is more applicable to your use case.

If it's stored somewhere online accessible without credentials, you can just wget or git cloneand proceed to the installation as usual.

I use quite a bit of multi-stage builds to reduce the size of the final image, i.e.,

FROM ubuntu as intermediate
COPY src.zip file.zip
RUN unzip file.zip
# proceed to installation

FROM ubuntu
COPY --from intermediate /build-output/ /usr/bin/

When your image is getting large, this is an elegant way to do it.

Hope it helps.

[SaltedPork]

Yep, it sure does help, thanks.