samtools sort: failed to create temporary file: No space left on device

Question

[lizhichao]

Hi,I run the cwl workflow in the local machine. I write the cwl and run in [/Bigdata], but the running work seems to fill the /dev and reports that No space left on device. I want to konw why？ my fastq is about 70G

   Filesystem      Size  Used Avail Use% Mounted on
udev             32G  4.0K   32G   1% /dev
tmpfs           6.3G  1.1M  6.3G   1% /run
/dev/vda1        99G   94G     0 100% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none             32G     0   32G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/vdc1       1.4T 1007G  302G  77% /var/lib/docker
/dev/vdb1       296G   49G  232G  18% /home
overlay         1.4T 1007G  302G  77% 
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdd1       2.0T  220G  1.7T  12% /Bigdata

Answer 1

[Tom]

Your cwl-runner is writing everything to a temporary directory before collecting the final output and placing it in /Bigdata.

The solution depends on which cwl-runner you are using. For cwltool, you could use the command line parameters --tmpdir-prefix and --tmp-outdir-prefix and specify a device with lots of room.

Depending on your os you could probably also bind the tmp directory to another drive with a command like mount -B /tmp /mnt/tmp.

[lizhichao]

Tank you very much! I believe this can solve my problem. I will try it as soon as poosible

Answer 2

[WouterDeCoster]

Probably because of a bunch of temporary files going to /tmp/. Samtools has a -T argument to specify where temporary files should be made.

[Carambakaracho]

Wouter is absolutely right. Just for you know what to look for next time, this most likely where your /temp is:

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        99G   94G     0 100% /

You're analysis shows you it's completely filled up. Something like this is always a warning sign and is very likely to cause system instability.

[lizhichao]

Thank you. my error is : samtools sort: failed to create temporary file "/BOredq/NA12878.sorted.bam.tmp.0000.bam": No space left on device And samtools sort module's "-T" option just specify the prefix instead of output dir. $samtools sort Usage: samtools sort [options...] [in.bam] Options: -l INT Set compression level, from 0 (uncompressed) to 9 (best) -m INT Set maximum memory per thread; suffix K/M/G recognized [768M] -n Sort by read name -t TAG Sort by value of TAG. Uses position as secondary index (or read name if -n is set) -o FILE Write final output to FILE rather than standard output -T PREFIX Write temporary files to PREFIX.nnnn.bam --input-fmt-option OPT[=VAL] Specify a single input file format option in the form of OPTION or OPTION=VALUE -O, --output-fmt FORMAT[,OPT[=VAL]]... Specify output format (SAM, BAM, CRAM) --output-fmt-option OPT[=VAL] Specify a single output file format option in the form of OPTION or OPTION=VALUE --reference FILE Reference sequence FASTA FILE [null] -@, --threads INT Number of additional threads to use [0]

can we specify all of cwl output dir including tmp files to running dir such as [ /Bigdata] ?

[WouterDeCoster]

And samtools sort module's "-T" option just specify the prefix instead of output dir.

And what if you would use, as a "prefix" /Bigdata/tmp/mytemp?

[lizhichao]

I dont think it could work，because the running environment is virtual after staging files. If i try to specify the external path such as /Bigdata/... in command,I believe it will failed because of not finding the path.