How to efficiently/quickly merge ~500k vcfs?

Question

[Lei]

I have a frankly ludicrous number of single-sample vcf.gz files (with their tabix) that I want to merge into on big file. I've previously used bcftools merge on 48 threads to merge 1000 and it took 15+ minutes. I'm pretty sure that time to complete won't scale linearly once I increase the number of samples to 500k+. Any suggestions? Should I merge groups of samples at a time like going up a tree? Should I use a different toos?

[Pierre Lindenbaum]

How to merge 20K single-sample VCFs *without* using plink or plink2?

[cmdcolin]

just saw there is a 'virtual codeathon' for "scaling vcf to millions of samples" soon, can sign up here https://ncbiinsights.ncbi.nlm.nih.gov/event/vcf-for-population-genomics-codeathon/

Answer 1

[Jeremy Leipzig]

I would suggest TileDB-VCF, which enables downstream analysis (and export) without the need to fire up a Spark cluster. (Disclaimer: I work for TileDB)

[Lei]

Interesting! I looked into TileDB-VCF a couple of months back and it looks like the tutorial has much improved! I'll give it a try as well.

[Jeremy Leipzig]

Feel free to reach out to me directly. I can walk you through some notebooks and/or provide some free credits to get you started.

Answer 2

[DBScan]

Do you have VCFs or gVCFs? For gVCFs you could also use HAIL (https://hail.is/) or GLNexus (https://github.com/dnanexus-rnd/GLnexus).