Nextflow workflow errors when run from Scratch dir?

Question

[Eliveri]

I am running a nextflow workflow on a HPC. The workflow is able to run fine when working from my home directory. I am running with NXF_VER=22.11.0-edge nextflow run main.nf -profile sge,apptainer

I copied the workflow and its files to my scratch directory. However I am now encountering errors (mostly in terms of finding files?).

The following process was running fine before are now causing issues (another related issue) when I run the workflow from the scratch directory rather than home directory:

 htsjdk.samtools.util.RuntimeIOException: java.nio.file.NoSuchFileException: /scratch/workflow/tmp/sortingcollection.5367666740078833193.tmp
    at htsjdk.samtools.util.SortingCollection.spillToDisk(SortingCollection.java:267)
    at htsjdk.samtools.util.SortingCollection.add(SortingCollection.java:182)
    at htsjdk.samtools.SAMFileWriterImpl.addAlignment(SAMFileWriterImpl.java:202)
    at htsjdk.samtools.AsyncSAMFileWriter.synchronouslyWrite(AsyncSAMFileWriter.java:36)
    at htsjdk.samtools.AsyncSAMFileWriter.synchronouslyWrite(AsyncSAMFileWriter.java:16)
    at htsjdk.samtools.util.AbstractAsyncWriter$WriterRunnable.run(AbstractAsyncWriter.java:123)
    at java.lang.Thread.run(Thread.java:750)
  Caused by: java.nio.file.NoSuchFileException: /scratch/workflow/tmp/sortingcollection.5367666740078833193.tmp
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
    at java.nio.file.Files.newByteChannel(Files.java:361)
    at java.nio.file.Files.createFile(Files.java:632)
    at java.nio.file.TempFileHelper.create(TempFileHelper.java:138)
    at java.nio.file.TempFileHelper.createTempFile(TempFileHelper.java:161)
    at java.nio.file.Files.createTempFile(Files.java:852)
    at htsjdk.samtools.util.IOUtil.newTempPath(IOUtil.java:396)
    at htsjdk.samtools.util.SortingCollection.newTempFile(SortingCollection.java:278)
    at htsjdk.samtools.util.SortingCollection.spillToDisk(SortingCollection.java:249)

I think this --TMP_DIR ${PWD}/tmp may be causing the issue

process sam_sort {

    tag "sam sorting ${pair_id}"
    label 'big_mem'

    publishDir "${params.outdir}/$pair_id"

    input:
    tuple val(pair_id), path(sam)

    output:
    tuple val(pair_id), path("${pair_id}.sorted.dup.bam"),
    path("${pair_id}.sorted.bam"),
    path("${pair_id}.bam"),
    path("${pair_id}.clean.bam"),
    path("${pair_id}_dup_metrics.txt")

    """
    # sam file sorting
    gatk --java-options "-Xmx${params.gatk_memory}g -Xms${params.gatk_memory}g" SamFormatConverter -R $ref -I ${pair_id}.sam -O ${pair_id}.bam

    gatk --java-options "-Xmx${params.gatk_memory}g -Xms${params.gatk_memory}g" CleanSam -R $ref -I ${pair_id}.bam -O ${pair_id}.clean.bam

    gatk --java-options "-Xmx${params.gatk_memory}g -Xms${params.gatk_memory}g" SortSam -R $ref -I ${pair_id}.clean.bam -O ${pair_id}.sorted.bam -SO coordinate --CREATE_INDEX true --TMP_DIR ${PWD}/tmp

    gatk --java-options "-Xmx${params.gatk_memory}g -Xms${params.gatk_memory}g" MarkDuplicates -R $ref -I ${pair_id}.sorted.bam -O ${pair_id}.sorted.dup.bam -M ${pair_id}_dup_metrics.txt -ASO coordinate --TMP_DIR ${PWD}/tmp
    """
}

config file:

profiles {

    apptainer {
        conda.enabled           = false
        apptainer.enabled       = true
        apptainer.autoMounts    = true
        docker.enabled          = false
        process.container       = 'file://nf-wgs-dsl2.sif'
    }
    sge {
        process {
            executor        = "sge"
            scratch         = true
            stageInMode     = "copy"
            stageOutMode    = "move"
            errorStrategy   = "retry"
            clusterOptions = '-S /bin/bash -o job.log -e job.err'
        }
        executor {
            queueSize = 1000
        }
    } 

}

Answer 1

[Carambakaracho]

Hi Eliveri,

From what I can tell your assumption is correct and ${PWD}/tmp is the problem, provided ${TMP} is /scratch/workflow. Probably all you miss is a mkdir tmp. Fmpov, I would simply write it in the directory of the process (maybe into a tmp subdirectory - including a mkdir tmp if you fancy).

Btw, there's a few things you can improve, so you can better make use of Nexflow's data flow control:

• Separate the commands into four processes
• Chain outputs and inputs

Cheers!

[Eliveri]

Thank you for the your suggestions. I am working on separating the commands into four processes. That is a great idea!

Answer 2

[colindaven]

Try it with just tmp, not ${PWD}/tmp

As you probably know, Nextflow handles files gracefully with lots of linking between inputs and outputs in the work/ directory by default. It's best to let Nextflow handle these paths, and not get in the way providing your own path logic.

Also as Carambakaracho says, separating into four processes will be much more performant and elegant.

[Eliveri]

Hi. Thank you. tmp solved the issue at hand.