Tool:Fast, flexible grep for very large FASTQ files

Question

[milesforjazz]

Got a very large FASTQ file that you want to quickly filter for the presence or absence of multiple target sequences?

I'm happy to release v.1.1.4 of grepq, a fast and flexible way of grepping large FASTQ files.

Free, open-source and built in Rust.

Crates.io

https://crates.io/crates/grepq

GitHub

https://github.com/Rbfinch/grepq

[jared.andrews07]

Might be worth adding a few of the tidbits from your README that emphasize the benefits of your tool over typical grep.

Answer 1

[milesforjazz]

grepq v1.1.6 is more than 1000 times faster than the venerable grep, and 11 times faster than ripgrep. Both grep and ripgrep, being general purpose tools, will search the entire file, including the quality scores. Occassionally, targets will 'lurk' in quality scores, leading to false positives.

grepq also has a tune subcommand to analyze matched target strings and update the number and/or order of regex patterns in your pattern file according to their matched frequency. This can further speed up the filtering process.

Check out the tool and README here:

https://crates.io/crates/grepq

If you've downloaded a previous version, updating is as simple as:

cargo install grepq

Answer 2

[milesforjazz]

grepq v.1.1.9 released today, bringing JSON support for pattern file input and tune subcommand output, allowing named regex sets and named regex patterns. gzip support was added in v.1.1.8

Check out grepq and README here:

https://crates.io/crates/grepq