Java.Io.Filenotfoundexception ( Too Many Open Files ) :Picard Tools

Question

[hellbio]

Hi,

I am using the following command,

java -jar /csc/lohi/dog_tools/picard-tools-1.70/AddOrReplaceReadGroups.jar \
  I=bull2.bam \
  O=bull2_rg_sorted.bam \
  SORT_ORDER=coordinate \
  RGID=bull2 \
  RGLB=bull2 \
  RGPL=Illumina \
  RGPU=bull2 \
  RGSM=bull2 \
  VALIDATION_STRINGENCY=SILENT \
  CREATE_INDEX=True \
  TMP_DIR=`pwd`/tmp

and it says

Exception in thread "main" net.sf.samtools.util.RuntimeIOException: java.io.IOException: (Too many open files)

Any suggestions to fix this?

Answer 1

[Biomonika (Noolean)]

Add the parameter named MAX_FILE_HANDLES_FOR_READ_ENDS_MAP, I set it to 1000 and it helped :-) with the explanation here

[hellbio]

MAX_FILE_HANDLES_FOR_READ_ENDS_MAP is a command line parameter for MarkDuplicates function whereas my task is AddOrReplaceReadGroups.jar which doesn't have the parameter.

Any help?

[david.ries84]

There is even another parameter for that. By looking at the link posted by Noolean you'll find:

There are several things you can do: 1) On Unix systems, the allowed number of open files can be increased. ulimit -n will show you how many open files a process is allowed to have. You can ask your system administrator to increase that number. 2) Many Picard programs use temporary files to sort. You can increase the value of the MAX_RECORDS_IN_RAM command-line parameter in order to tell Picard to store more records in fewer files in order to reduce the number of open files. This will, however, increase memory consumption. 3) MarkDuplicates has the command-line parameter MAX_FILE_HANDLES_FOR_READ_ENDS_MAP. By reducing this number, you reduce the number of concurrently open files (trading off execution speed).

[ravi.uhdnis]

Thank you for this information. I was struggling with Picard's AddOrReplaceReadGroup in order to sort and add Read Group information while converting my .sam to .bam file. ulimit -n (500,000) didn't helped but then I used MAX_RECORDS_IN_RAM=1000000, which finally enabled my command to run successfully and I got the desired output .bam file.

[lzwright]

I was having problem with MarkDuplicates and setting MAX_FILE_HANDLES_FOR_READ_ENDS_MAP to 1000 as pointed out above does do the trick!

[Biomonika (Noolean)]

Great, I will be happy if you upvote my answer in case it helped you.

Answer 2

[Istvan Albert]

Raise your ulimit for files:

http://posidev.com/blog/2009/06/04/set-ulimit-parameters-on-ubuntu/

[hellbio]

Hi Albert,

ulimit is set to unlimited and MAX_RECORDS_IN_RAM=1000000 but, I still have the same issue. Any suggestions would help.