Error: SortDetails.cpp, 1111: Unable to create FILE* for temp file: No space left on device. Out of memory.

Question

[mropri]

I am receiving the following error when I try to use bedmap to get the coverage of specific coordinates in a bed file:

Error: SortDetails.cpp, 1111: Unable to create FILE* for temp file: No space left on device. Out of memory.

I understand that I have to free up space somewhere in my files, don't know exactly where. Also, I am on a university network so do not have admin privileges to use commands like sudo -apt clean to remove cache or whatever files are getting stored where there is little memory. The folder I am using is our labs folder and has plenty of space (10 TB), so confused why the error is there. Appreciate any help!

My df -h output is shown here:

https://docs.google.com/presentation/d/1BlnrlnXxbjpK2BsgPqNcmaetmhWhoON1omlv4tm7D9k/edit?usp=sharing

[Ram]

Your /tmp is out of space. Unless bedmap allows setting a different temp location (or accepts the value in $TMPDIR, and thus lets you change the temp location), you'll need to clean up your temp location.

On second thought, your /tmp only shows a use of 1%, so my suggestion might not really apply

[mropri]

Is there a way to set a different temp location?

[Ram]

If the manual does not provide an option for this, try setting TMPDIR to a directory under $PWD like so:

OLD_TMPDIR=$TMPDIR
mkdir temp_dir
export TMPDIR=temp_dir
bedmap ....

If the command runs to completion successfully,

export TMPDIR=$OLD_TMPDIR
unset OLD_TMPDIR

[Alex Reynolds]

This will not solve this particular problem, as sort-bed does not use the POSIX tempnam function (as it is not entirely safe to use on a multiuser system), and so it does not use TMPDIR. It would be a useful feature enhancement, but it is easy enough to specify --tmpdir and --max-mem.

[Ram]

Sorry, I was looking at bedmap, not sort-bed. bedmap doesn't seem to have a tmp-dir option.

[Alex Reynolds]

I don't think that it would. I don't know why that error would come out of bedmap. If that's the case, I'd love to see what command is being sent.

[mropri]

Just wanted to say thank you guys for your help in this, Alex I know you had initially helped me with getting raw reads of my coordinates in a bed file from my bam file. Always so thankful for the time you guys give for detailed answers!

[mropri]

Thank you Ram for your help in this. Really appreciative of the help you give us all in helping troubleshoot errors we encounter.

Answer 1

[Alex Reynolds]

To sort data with a custom temporary directory, specify use sort-bed --tmpdir <dirname> --max-mem <value> unsorted.bed.

The dirname passed to --tmpdir will be where sorted chunks of the input are stored temporarily, before the final sort.

Note the use of --max-mem. The value passed to --max-mem specifies the maximum system memory that will be used (e.g. 2G for two gigabytes of RAM, if available). This is not disk space. Use free -m to get a sense of how much free system memory you have available.

The temporary directory passed to --tmpdir will need sufficient free space to temporarily store a partitioned duplicate of your input file.

So if /tmp doesn't have enough free space to store a copy of your input BED file (your Google Doc suggests it is 4GB in size, and some of it may be used by other processes and users), you would use a different directory or share, like a subdirectory of /network/rit/home/ed717179, where it looks like you have plenty of free disk space.

[mropri]

Thank you so much, going to try this out. Truly, appreciate all your help.

[Ram]

If Alex's answer solved your question, please use the green check mark on the left to mark the post as resolved.

[mropri]

Sorry, but trying Alex's method, will hopefully soon check the mark.

[mropri]

Alex you had helped me out with another question of mine where I was looking to find coverage of my bed file using the associated bam file. So the command I am using is the following:

bedmap --skip-unmapped --delim '\t' --echo --count --bases-uniq --echo-ref-size --bases-uniq-f CombinedSE.sorted.bed <(bam2bed < AT1_2_H3K.sorted.bam) > AT1_2_H3K.coverage.bed

Where CombinedSE.bed is the coordinates of all the super enhancers defined by the ROSE algorithm in my samples. I am using one of my samples bam files to get raw counts for those specific coordinates, but I keep getting the error that is originally listed in the question. The solution you provided above, would I be able to tack that on to the command I am using already to get coverage to allocate a subdirectory to temporarily store the inputs? I feel really bad about bothering you over and over, but appreciate your help.

[Alex Reynolds]

I suspect it is this part that is causing the disk space error:

<(bam2bed < AT1_2_H3K.sorted.bam)

The program underneath bam2bed uses sort-bed under the hood. It offers two options that configure sort-bed in the way I describe in my first answer:

bam2bed --max-mem=[value] --sort-tmpdir=[dirname] < reads.bam

So, basically, you would add those options to the bam2bed portion of <(bam2bed < AT1_2_H3K.sorted.bam):

... <(bam2bed --max-mem=[value] --sort-tmpdir=[dirname] < AT1_2_H3K.sorted.bam) ...

Replace value and dirname, as needed.

Depending on the particulars of your BAM file, other parameters may be useful. Take a look at bam2bed --help or the online documentation, to look at the BAM-specific conversion options. For instance, using --reduced with bam2bed can speed things up quite a bit for coverage measurements.

[mropri]

Thank you so much Alex, this command worked and I was able to get coverage of the bed file using the corresponding bam file. Appreciate your time and help.

[mropri]

Ok, will do, thank you so much for your time in helping me.