error in MarkDuplicates Picard

Question

[evelyn]

Hello,

I have aligned multiple fastq files using bwa-mem and got .sorted.bam files:

INPUT_DIR=/path/trimmed
OUTPUT_DIR=/path/result
INPUT_FILE_ONE=$(ls -1 $INPUT_DIR/*_R1_paired.fastq.gz | sed -n ${RUN}p)
SAMPLE=$(basename "$INPUT_FILE_ONE" _R1_paired.fastq.gz)

bwa mem genome.fasta ${INPUT_DIR}/${SAMPLE}_R1_paired.fastq.gz ${INPUT_DIR}/${SAMPLE}_R2_paired.fastq.gz > ${OUTPUT_DIR}/${SAMPLE}_paired_bwa.sam
samtools view -S -b ${OUTPUT_DIR}/${SAMPLE}_paired_bwa.sam > ${OUTPUT_DIR}/${SAMPLE}_paired_bwa.bam
samtools sort ${INPUT_DIR}/${SAMPLE}_paired_bwa.bam -o ${OUTPUT_DIR}/${SAMPLE}_paired_bwa.sorted.bam
samtools index ${OUTPUT_DIR}/${SAMPLE}_paired_bwa.sorted.bam

Then I tried using Picard to remove duplicates from sorted.bam files using:

INPUT_DIR=/path/result
OUTPUT_DIR=/path/Duplicate_marking_picard
INPUT_FILE_ONE=$(ls -1 $INPUT_DIR/*_paired_bwa.sorted.bam | sed -n ${RUN}p)
SAMPLE=$(basename "$INPUT_FILE_ONE" _paired_bwa.sorted.bam)
echo "RUN #${RUN} with sample ${SAMPLE}"

java -Xms1g -Xmx3g -jar picard.jar MarkDuplicates \
I=${INPUT_DIR}/${SAMPLE}_paired_bwa.sorted.bam \
O=${OUTPUT_DIR}/${SAMPLE}_picard.sorted.bam \
M=${OUTPUT_DIR}/${SAMPLE}_metrics.txt \
TMP_DIR=`pwd`/tmp

However, after resulting in .picard.sorted.bam and metrics.txt files for input files, it started resulting in only .picard.sorted.bam and no metrics.txt file. When I checked the log files for such cases, it gives a long message including this error:

Exception in thread "main" htsjdk.samtools.util.RuntimeIOException: Write error; BinaryCodec in writemode; streamed file (filename not available)
Caused by: java.io.IOException: Disk quota exceeded

I tried again but I still got the same error after getting results for some files. Thank you for the help!

[h.mon]

Disk quota exceeded

This has nothing to do with Picard, you ran out of space assigned to you by the system administrators. Either you have to ask for more space, or delete unneeded files.

You could save a lot space by streaming the bwa mapping directly into samtools:

bwa mem genome.fasta ${INPUT_DIR}/${SAMPLE}_R1_paired.fastq.gz ${INPUT_DIR}/${SAMPLE}_R2_paired.fastq.gz \
  | samtools sort -o ${OUTPUT_DIR}/${SAMPLE}_paired_bwa.sorted.bam -

[evelyn]

Thank you! I have checked our group's space and we still have enough space left. We did not get any space related notification which we usually get once we use 90% of our assigned space. I needed .sam files that's why I did not choose to skip that.

I am wondering if it has to do with the same TMP_DIR in picard command. I just checked and the TMP_DIR is empty and I am not sure if I have used it correctly in picard command line.

[GenoMax]

TMP_DIR may indeed be empty when there is no job running. You can re-start the job and watch that directory. You may have a separate quota on the directory where you have TMP_DIR.

[evelyn]

Thank you! It was empty while the job was running. That's why I am wondering if my code is correct to use TMP_DIR. How can I assign a separate quota on this directory.

[GenoMax]

If that did not work then you could try -Djava.io.tmpdir=/directory_path.

BTW did you make a directory called tmp in your working directory when using pwd/tmp.

[evelyn]

Thank you! Yes, I made a directory called tmp in my working directory. I will try your suggestion.

[evelyn]

This command did not work. I get the same error after some samples.

[MatthewP]

can you post quota -s command result?

[evelyn]

Our group quota is 11118938420 kbytes used out of 13958643712 kbytes available. I submitted the job by increasing the memory. I am not sure if it will work.