Nextflow: No space left on device, process using output directory as input?

Question

[Eliveri]

The last process of my Nextflow workflow failed with the following job.log. The workflow is run with a singularity container.

cp: error writing 'results/FILE_ST01_WGS_S1/S3TIMULATE_ST01_WGS_S1.sam': No space left on device
cp: cannot create regular file 'results/FILE_ST01_WGS_S1/FILE_ST01_WGS_S1.clean.bam': No space left on device
cp: cannot create regular file 'results/FILEST01_WGS_S1/FILE_ST01_WGS_S1.sorted.bam': No space left on device
cp: cannot create directory 'results/FILE_ST08_WGS_S8': No space left on device
...

It is a bit strange, since looking in the results directory, I could find all the directories and files which were created in previous processes. And all the previous Nextflow processes were completed without error (no errors in job.log(s)). The last process is also not very memory intensive. It is an rscript with takes some of the .txt and .tsv results from the outdir and makes a report.

There seems to be adequate space on the drive (it is a scratch drive).

The last process is to run a report which has little to do with the files and directories mentioned in the job.log errors.

The last process:

process run_report {

    tag "Run quality report"

    publishDir params.outdir, mode:'copy'

    input:
    tuple path(file_1), path(file_2)
    path outdir
    path rscript

    output:
    file('run_quality_report.html')

    script:
    """ 
    Rscript -e 'rmarkdown::render(input = "$rscript", output_dir = getwd(), params = list(directory = "$outdir"))'
    """
}

After inspecting the .command.run. It seems the process might be trying to copy ALL of the files in the outdir because outdir is an input in the 'nxf_stage()could this be the issue? And should I avoid using theoutdir` as an input?

nxf_stage() {
    true
    # stage input files
    rm -f Bam_stats_pf_Final.tsv
    rm -f Bam_stats_hs_Final.tsv
    rm -f results_victoria_all
    rm -f run_quality_report.Rmd
    cp -fRL /wynton/scratch/finterly/new_workflow/work/fe/ec06795d4f7811a7385efbc69cf4fa/Bam_stats_pf_Final.tsv Bam_stats_pf_Final.tsv
    cp -fRL /wynton/scratch/finterly/new_workflow/work/ce/9c20df52f840bb21f13c48dc4df75d/Bam_stats_hs_Final.tsv Bam_stats_hs_Final.tsv
    cp -fRL /wynton/scratch/finterly/results_victoria_all results_victoria_all
    cp -fRL /wynton/scratch/finterly/new_workflow/run_quality_report.Rmd run_quality_report.Rmd
}

nxf_unstage() {
    true
    cp .command.out /wynton/scratch/finterly/new_workflow/work/f5/85eaf31ed30707d936a7fe8a8c9841/.command.out || true
    cp .command.err /wynton/scratch/finterly/new_workflow/work/f5/85eaf31ed30707d936a7fe8a8c9841/.command.err || true
    cp .command.trace /wynton/scratch/finterly/new_workflow/work/f5/85eaf31ed30707d936a7fe8a8c9841/.command.trace || true
    [[ ${nxf_main_ret:=0} != 0 ]] && return
    IFS=$'\n'
    for name in $(eval "ls -1d run_quality_report.html" | sort | uniq); do
        nxf_fs_move "$name" /wynton/scratch/finterly/new_workflow/work/f5/85eaf31ed30707d936a7fe8a8c9841 || true
    done
    unset IFS
}

[colindaven]

Yes, don't use the outdir as an input. You might get infinite loops.

Answer 1

[ATpoint]

That's not a nextflow error. Check whether there are per-user quotas that you might exceed on scratch. Total scratch size is often not a good metric. Check with the admins and documentation of that server.

[Eliveri]

Thank you. I will check with admins about per-user quota.

Answer 2

[Mensur Dlakic]

I often tell people that computers have no sense of humor. The error message you got is not a prank - that is what the computer sees. So either you are wrong in what space is available (or available to you, to be more precise) in the current directory, or the program is using a temporary directory somewhere else that has ran out of space.

[Eliveri]

Thank you! You are correct, I took a closer look at the .command.sh in particular. It seems that by using the outdir as input, I may be causing nextflow to copy everything?

[Maxime Garcia]

Nextflow should only link inputs (unless stated otherwise), so even if you use outdir as input it should not be much of an issue, but still I'd recommend not to do so, and properly select the files or folder from the previous process that you need for an input as the next process