HPC settings for concatenating large VCF.GZ files

Question

[Katherine]

Dear all,

First-time poster here. :)

I was hoping to ask for your recommendations on what to request on a HPC to concatenate ~22 chromosomes (each vcf.gz files is ~15GB, so ~330GB in total).

e.g. with bcftools concat -Oz chr1.vcf.gz chr2.vcf.gz ... chrX.vcf.gz > allchr.vcf.gz

Could I request 32 CPUs each with 16GB of memory (512GB)? Would that work?

Any suggestions at all would be appreciated!!

Thanks in advance

[LauferVA]

Katherine - There are many ways to resolve that question, none of which should require that much dedicated memory. Irrespective, I really encourage you to tabix index your vcf files. Once this index file is created (which only needs to be done once), you will be able to perform operations that would have taken minutes in seconds, including operations similar to what you propose. VAL

Answer 1

[4galaxy77]

bcftools concat stores only a small stream of data in memory. Therefore, you would need, at most, 1Gb of memory to perform this operation and one cpu core. You can request multiple threads to perform multithreaded gzipping of the output though.

[Katherine]

Thanks for this information @4galaxy77 ! How many hours would you expect a job to run using bcftools to concatenate just TWO vcf.gz files (~15GB each)? At the moment, this job has been running for 20hrs. Thanks in advance, Katherine.