speeding up bcftools view

Question

[eb13]

Hi all - I have a very large multi sample vcf file which I am trying to subset by a list of sample IDs, however, my current approach is working very slowly (>2hr per chromosome) and I am wondering if there are any tricks to making it run faster with large files? Here is my current approach:

for file in /vcffiles/*.vcf.gz; do
    bcftools view -Oz -S sample_list.txt $file > /output/subset_"${i##*/}" 
done

Thanks in advance for any suggestions!

[mohammadhassanj]

Maybe this link is useful: How to parallelize bcftools mpileup with GNU parallel?

[eb13]

thank you for your helpful responses!

Answer 1

[barslmn]

Another solution with tsp and background processes.

# this sets number of max jobs. Here we use the number of processes. You might want to change this to another number.
tsp -S $(nproc)

# rest is similar. we just add tsp to start of the command and & at the end.
# & at the end calls all the processes at once but tsp queues them and calls them in batches.
for file in /vcffiles/*.vcf.gz; do
    tsp bcftools view -Oz -S sample_list.txt $file > /output/subset_"${i##*/}" &
done

Answer 2

[Pierre Lindenbaum]

let's do it using nextflow, I won't test it so there will be some small bugs, but you get the idea.