merge large amount of fastq files into a single one

Question

[catherine]

I have 30 small fastq files from same sample, and I want to merge it into one file. I know the command is

cat file1.fastq file2.fastq > bigfile.fastq

but is there any short cut for doing it? It just looks silly to type 30 file names one by one...

Thank you for any idea!

[BioApps]

Those with Windows can use this GUI tool (works also on Linux via wine): http://www.dnabaser.com/download/Merge%20Fasta/index.html

Answer 1

[David Langenberger]

cat file*.fastq > bigfile.fastq

[catherine]

οh yeah! i was so stupid!

[Brian Bushnell]

Be cautious about this approach! Depending on your system, you can enter an endless loop of concatenating the new file to itself. I strictly do:

cat *.fq > merged.fastq** or **cat *.fastq > merged.fq

...or whatever is needed to ensure the pattern does not match the new file being created.

[lh3]

Does this happen? My understanding is that shell first parses "*.fq" and at that time "merged.fq" has not been generated yet. I bet a lot of people must have typed "cat *.txt > out.txt". Shell developers should have been aware of such an issue for many years. I could be wrong, though.

[David Langenberger]

Actually, it happened to me once. That's why I put the 'file' as prefix for the input and 'bigfile' for the output. But I didn't know that it is system dependent. Thanks for mentioning it, Brian.

[lh3]

I was wrong. You and Brian are right. I can reproduce this endless loop.

Answer 2

[Pierre Lindenbaum]

It just looks silly to type 30 file names one by one...

With file globbing

cat file*.fastq > bigfile.fastq

Note: It also works with fastq.gz files. (http://stackoverflow.com/questions/8005114)

cat file*.fastq.gz > bigfile.fastq.gz

[Bioinfonext]

Error while using: cat*.R1_unmapped.fq  > unmapped_R1.fq

216_7W_Ca1_R1_unmapped.fq  
216_9W_Co2_R1_unmapped.fq 
 218_5W_Pa1_R1_unmapped.fq  
218_7W_Pa2_R1_unmapped.fq  

[root@psgl unmapped]# cat *.R1_unmapped.fq  > unmapped_R1.fq\

cat: *.R1_unmapped.fq: No such file or directory

[vmicrobio]

(extra dot)

cat *_R1_unmapped.fq > unmapped_R1.fq

[BioApps]

Nice solution. Yes. Basically you need to do a 'dumb' file merge.