remove empty reads from fastq files

Question

[dentepre]

Hi,

Before perform STAR mapping I need to remove empty reads from my fastq files.

How can I do it without using bioawk (I cannot add it to the HPC). I have bedtools, samtools and bbmap, but I didn't find any solution in these packages.

Best, d.

[ATpoint]

Why can't you edit to your cluster? It is straight-forward to compile:

git clone https://github.com/lh3/bioawk.git
cd bioawk
make

or use a package manager like miniconda for it. You will have a hard time working on a HPC (or any machine) if you cannot add any software in the long-term.

[cpad0112]

try cutadapt minimum length option or seqkit seq --min-len option

Answer 1

[GenoMax]

With BBMap suite:

reformat.sh in=your.fq.gz out=filtered.fq.gz minlength=N

Set N to a reasonable number (very short reads are not useful anyway). Set to 1 if you just want to remove empty reads.

[WAGNER]

how would the command look for two input files?

[GenoMax]

I assume you mean a paired-end dataset. For single-end data you will simply need to run this twice for the two files.

For paired-end data use

reformat.sh in1=R1.fq.gz in2=R2.fq.gz out1=R1.filtered.fq.gz out2=R2.filtered.fq.gz  minlength=N

Answer 2

[Pierre Lindenbaum]

gunzip -c input.fq.gz | paste - - - - | awk -F '\t' '($2!="")' | tr "\t" "\n"