vcftools - Error: Could not open temporary file.

Question

[Figaro1993]

Hello,

there is an error when I try to filter a VCF document for MAF. Here is a part of the the vcf-file:

#CHROM POS        ID REF ALT QUAL FILTER INFO FORMAT Geno_10886
 2:  chr1H 523 chr1H:523   C   A    .   PASS    .     GT       0|0
 1:  chr1H 445 chr1H:445   C   T    .   PASS    .     GT       0|
 2:  chr1H 523 chr1H:523   C   A    .   PASS    .     GT       0|0
 3:  chr1H 535 chr1H:535   G   A    .   PASS    .     GT       0|0
 4:  chr1H 587 chr1H:587   A   G    .   PASS    .     GT       0|0
 5:  chr1H 757 chr1H:757   C   A    .   PASS    .     GT       0|0

This is the command I'd like to execute:

~/vcftools_0.1.13/bin/vcftools --vcf prepared_chr_1.vcf --maf 0.05 --plink --out ~/input/strict.filtered.gwas.chr1

However, I get this error:

VCFtools - v0.1.13
(C) Adam Auton and Anthony Marcketta 2009

Parameters as interpreted:
    --vcf prepared_chr_1.vcf
    --maf 0.05
    --out /proj/input/strict.filtered.gwas.chr1
    --plink

After filtering, kept 1315 out of 1315 Individuals
Writing PLINK PED and MAP files ... 
Error:  Could not open temporary file.

Do you have any suggestions? If I only use

~/vcftools_0.1.13/bin/vcftools --vcf prepared_chr_1.vcf --maf 0.05

without the --plink --out options, It seems to work but the output is of course not written into a file.

Thank you in advance for your answers!

Max

[Pierre Lindenbaum]

vcftools is deprecated; use bcftools view --min-af/--max-af

[zx8754]

Plink can read VCF files, no need to convert: https://www.cog-genomics.org/plink/1.9/input#vcf

[Figaro1993]

It appears that my vcf file is actually readable. If I use the following command, it works. However, I need PLINK PED and MAP files...

~/vcftools_0.1.13/bin/vcftools --vcf prepared_chr_1.vcf --maf 0.05 --out ~/input/strict.filtered.gwas.chr1

VCFtools - v0.1.13 (C) Adam Auton and Anthony Marcketta 2009

Parameters as interpreted: --vcf prepared_chr_1.vcf --maf 0.05 --out /proj/input/strict.filtered.gwas.chr1

After filtering, kept 1315 out of 1315 Individuals After filtering, kept 1443575 out of a possible 2848679 Sites Run Time = 244.00 seconds

[Figaro1993]

Our admins have set a limit of 4096 simultaneously open files on our cluster.

We used strace to track down the problem:

strace -e open,openat,read,connect ~/vcftools_0.1.13/bin/vcftools --vcf input.vcf --maf 0.05 --temp /tmp --plink  --out output.chr1

... (lots of similar lines) ....
openat(AT_FDCWD, "/tmp//vcftools.GiYYWq", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 1022
openat(AT_FDCWD, "/tmp//vcftools.4y5nep", O_RDWR|O_CREAT|O_EXCL, 0600) = 1023
openat(AT_FDCWD, "/tmp//vcftools.4y5nep", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 1023
openat(AT_FDCWD, "/tmp//vcftools.dyIiuq", O_RDWR|O_CREAT|O_EXCL, 0600) = -1 EMFILE (Too many open files)
Error:  Could not open temporary file.

+++ exited with 12 +++

The problem is that vcftools is opening a huge number of temporary files. Why does it need so many open files? Would it be possible to open less files and define some offsets within the files to reduce the number of file handles?