Memory Efficient Bedtools Sort And Merge With Millions Of Entries?

Question

[14134125465346445]

I would like to know if there is a memory-efficent way of sorting and merging a large amount of bed files, each of them containing millions of entries, into a single bed file that merges the entries, either duplicated or partially overlapping, so that they are unique in the file.

I have tried the following but it blows up in memory beyond the 32G I have available here:

find /my/path -name '*.bed.gz' | xargs gunzip -c | ~/src/bedtools-2.17.0/bin/bedtools sort | ~/src/bedtools-2.17.0/bin/bedtools merge | gzip -c > bed.all.gz

Any suggestions?

Answer 1

[Alex Reynolds]

Perhaps consider using BEDOPS sort-bed --max-mem to perform a sort within system memory (for example, --max-mem 24G, which asks for 24 GB of your host's 32 GB of system memory) and bedops --merge to calculate merged elements from sorted data:

$ find /my/path -name '*.bed.gz' -print0 \
    | xargs -0 gunzip -c \
    | sort-bed --max-mem 24G - \
    | bedops --merge - \
    | gzip -c \
    > answer.gz

In this example, the sort-bed --max-mem operation will run a quicksort on 24 GB chunks of data, and then apply a merge sort on each quicksort-sorted chunk. The bedops --merge operation will run fast (about 1/3rd the execution time of alternatives, which is a substantial savings for operations on data of this scale) and with a very low, constant memory profile on sorted input, but it will discard ID, score and strand data (if present) in calculating overlapping regions.

Answer 2

[brentp]

use the sort provided in your linux distribution.

... | sort -k1,1 -k2,2n | ...

that will write temporary files to disk as needed.

[Istvan Albert]

a minor addition to this, there is an -m option for the sort that takest files that are already individually sorted and merges them into one

[14134125465346445]

I think this would work if I wasn't gunzip'ing the files.

[14134125465346445]

I tried this version and it uses a very small amount of memory. It is slower than the equivalent bedtools sort, but it solves my problem.

Answer 3

[Ido Tamir]

merge first each individual bed file, then merge the merged files like:

for f in `find /my/path -name '*.bed.gz'`; do gunzip -c $f | ~/src/bedtools-2.17.0/bin/bedtools sort | ~/src/bedtools-2.17.0/bin/bedtools merge ; done | ~/src/bedtools-2.17.0/bin/bedtools sort | ~/src/bedtools-2.17.0/bin/bedtools merge | gzip -c > bed.all.gz

Answer 4

[e.rempel]

I had now good experience with mosdepth