Editing tools script in a docker container

Question

[Barry Digby]

Hi,

I have created a container using singularity, pulling an image from docker hub (here).

All i want to do is add a -T /data/scratch/ flag to samtools sort commands in two of the tools wrapper scripts. I have run sudo singularity shell xxx.simg and viewed the scripts, which are all -rw-r--r--. After applying chmod 777 to the files and applying the change, everything looks good. But my changes are discarded when I exit the container.

Any tips?

Thanks, ,

Barry

[steve]

you either re-build the Singularity container from the recipe (Dockerfile or Singularity file) with the changes you like or, preferably, do not use hard-coded locations to write files to, you should be using the volume bind flags to bind in locations on the host system and direct the container to read and write using those locations.

[Barry Digby]

I managed to do what I set out to do in the answer below, not sure if it is useful though.

My scripts are being run by nextflow, using containers for processes. Inspecting the .nextflow.log file, I've run into "[Errno 28] No space left on device:" errors which kill the nextflow process, and subsequent processes. Given the tool I am using is composed of several python scripts to (amongst other things) write BAM files for one step, and SAM files for another step in the same nextflow process, I thought setting a manual tempdir for samtools commands would help.

This is getting off topic for the post but might you be able to advise some trouble shooting tips?

Answer 1

[karl.stamm]

That's normal behavior for docker, no changes are saved by design. You can put the modifications into the scripts that run on startup, that way the changes are executed in each container anew. Using a script to modify the samtools sort commands may be tricky, so perhaps having it download a complete copy of the samtools script from a known location will be easier than line edits.

Answer 2

[Barry Digby]

Ok here goes.

The motivation for this was to specify where CIRIquant writes its temporary SAM | BAM files. I ran into disk space issues on my local cluster so wanted to specify a "/scratch" dir. All I had to do was edit the wrapper scripts that execute CIRIquant.

Pull image to edit

Pull image from docker:

sudo docker pull barryd237/ciriquant_v1.0.1

Check:

sudo docker image ls

REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
barryd237/ciriquant_v1.0.1   latest              1257a6f94f3c        30 hours ago        1.45GB

Currently just have an image. Need to shell into the container:

sudo docker run -it barryd237/ciriquant_v1.0.1:latest

Make whatever changes you need. Remeber to install nano or vim in the container to allow edits. Exit container.

Get ID of container you just made edits to:

sudo docker ps -a

CONTAINER ID        IMAGE                               COMMAND             CREATED             STATUS                          PORTS               NAMES
b61dde83d5bb        barryd237/ciriquant_v1.0.1:latest   "/bin/bash"         3 minutes ago       Exited (0) About a minute ago                       adoring_wiles

Commit the edited container:

sudo docker commit b61dde83d5bb ciriquant_v1.0.1_update

sudo docker image ls

REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
ciriquant_v1.0.1_update      latest              b4575687eb52        39 seconds ago      1.45GB
barryd237/ciriquant_v1.0.1   latest              1257a6f94f3c        30 hours ago        1.45GB

Push to Docker Hub

sudo docker login --username=barryd237

Use image ID to tag the edited docker image

sudo docker tag b4575687eb52 barryd237/ciriquant_update:push_update

REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
barryd237/ciriquant_update   push_update         b4575687eb52        9 minutes ago       1.45GB
ciriquant_v1.0.1_update      latest              b4575687eb52        9 minutes ago       1.45GB
barryd237/ciriquant_v1.0.1   latest              1257a6f94f3c        30 hours ago        1.45GB

Push to Docker Hub:

sudo docker push barryd237/ciriquant_update.

Build with Singularity

I am primarily a singularity user.. so the idea was to edit the image from Docker hub and build it in singularity.

singularity pull docker://barryd237/ciriquant_update:push_update

singularity shell -B $(pwd) ciriquant_update-push_update.simg

Check if changes were made:

# sort hisat2 bam
sorted_bam = '{}/{}.sorted.bam'.format(align_dir, prefix)
sort_cmd = '{} sort -T /data/bdigby/scratchspace/ --threads {} -o {} {}'.format(
    utils.SAMTOOLS,
    thread,
    sorted_bam,
    hisat_bam,
)

Happy Days.