how to unzip the files in batch?

Question

[flavobacteria]

Hi guys, I know this question is dumb. How to convert the .fastq.gz into fastq in batches? For example, if we have 100 .fastq.gz, how can we use the command to let them to be .fastq then? Thank you

[Giovanni M Dall'Olio]

Why do you need to unzip the fastq files? In most cases it is better to keep them compressed. Most NGS tools can handle compressed files directly, and it is generally faster to read a compressed file than an uncompressed one.

[Ashutosh Pandey]

Look at a few "xargs" usage on internet or simply try gunzip *.fastq.gz

[lh3]

If you have LSF (replace bsub with qsub -cwd for SGE):

ls *.fastq.gz | xargs -i echo bsub gzip -d {} | sh

Answer 1

[arnstrm]

GNU Parallel, FTW!

If you have access to a HPC cluster, open a interactive job session. In my case I have a 64 core node, so I do it as:

qsub -I -l nodes=1:ppn=64 -l walltime=1:00:00

then run gunzip on all 64 processors

parallel -j64 "gunzip {}" ::: *.fastq.gz

I'll be done in no time!

Answer 2

[venu]

$ gunzip *.gz

or

$ for f in *.gz; do gunzip $f; done

[5heikki]

For parallel processing (assuming more cores than jobs)

for f in *.gz; do gunzip $f & done

[GouthamAtla]

This would start as many jobs as the number of files, which is not an elegant way. Best way to do it is to use GNU Parallel as told by arnstm.

parallel --jobs <int cores> gunzip {} ::: *.fastq.gz

[jeremieaauger]

Awesome! Worked great on my 480 files

[dago]

Isn't this the easiest way? =)

Answer 3

[tomc]

Or don't.

pass them compressed if the tool handles it ,or decompress on the fly i.e.

zcat file.gz | whatever_tool

There are a bunch of utilities for processing gzip without explicitly writing out the unzipped file

http://www.nongnu.org/zutils/manual/zutils_manual.html

Answer 4

[Vivek Todur]

Hi,

PIGZ could be potential solution, its parallel version of GZip and uses the multi threading out of the box with so many other parameters to tweak the performance. You can find one here: http://zlib.net/pigz/

You can simply run pigz -d *.gz to extract the .gz files

Thanks

[Manu Prestat]

pigz is indeed a very simple way to achieve such a task.

Answer 5

[rtliu]

http://stackoverflow.com/questions/16038087/extract-all-gz-in-a-directory-linux

Answer 6

[ravi.uhdnis]

It might do the job

gunzip '*.gz'

Answer 7

[cvu]

gunzip *.fastq.gz will do the job!