Optimizing VCF File Merging Using bcftools

Question

[o.h3096]

Hello,

I am trying to merge 1200 WGS VCF files (around 9 GB each) using bcftools. I want to merge them by chromosome to make the output files easier to work with.

I am using an HPC cluster with 156 threads and 1to RAM, but I’m not sure how to optimize the resources to make the merging process faster. I used parallel but it didnt work as i get this error.

"Could not load local index file 'path/to/file.tbi' : Too many open files"

Even when I ran bcftools for a single chromosome, I got the same error message.

Any advice on how to resolve this issue and optimize resource usage would be greatly appreciated!

Thank you,

Answer 1

[GenoMax]

This may be related to ulimit option set for you account. See: https://askubuntu.com/questions/1182021/too-many-open-files

Answer 2

[Istvan Albert]

just so that there is also an answer here try something like as GenoMax points out in the link

ulimit -n 100000 

though the sys administrator may have some limits what you can set the limit to,

in that case merge in batches, merge 100 at the time per process.

now you have 12 files now merge those

Answer 3

[Pierre Lindenbaum]

see my answer using nextflow: How to merge 20K single-sample VCFs *without* using plink or plink2?

[o.h3096]

Thank you all! I adjusted the ulimit, and it worked perfectly.