How to combine pair end multiple fastq raw read file

Question

[Bioinfonext]

I want to combile two files into one file:

First files name is like this:

218_9W_Pa2_R2_unmapped.fq

218_9W_Pa2_R1_unmapped.fq

218_9W_Pa1_R2_unmapped.fq

218_9W_Pa1_R1_unmapped.fq

218_9W_Co2_R2_unmapped.fq

218_9W_Co2_R1_unmapped.fq

Second file names is like this:

218_9W_Pa2_R2_mapped.fq

218_9W_Pa2_R1_mapped.fq

218_9W_Pa1_R2_mapped.fq

218_9W_Pa1_R1_mapped.fq

218_9W_Co2_R2_mapped.fq

218_9W_Co2_R1_mapped.fq

After combined both files produce file like this:

218_9W_Pa2_R2.fq

218_9W_Pa2_R1.fq

218_9W_Pa1_R2.fq

218_9W_Pa1_R1.fq

218_9W_Co2_R2.fq

218_9W_Co2_R1.fq

Actually, I want to combine mapped and unmapped raw read file to produce final set

Answer 1

[shenwei356]

Shell:

for f in *_mapped.fq; do
    prefix=${f/_mapped.fq/};
    cat $f ${prefix}_unmapped.fq > $prefix.fq;
done

Or using rush (A GNU parallel like tool in Go surpporting Linux/OS X/Windows!)

ls *_mapped.fq | rush 'cat {} {.^_mapped}_unmapped.fq > {.^_mapped}.fq '

--- update ---

Much easier in rush v0.1.8+:

ls *_mapped.fq | rush -v p={.^_mapped}  'cat {} {p}_unmapped.fq > {p}.fq '

[Bioinfonext]

Hi,

Can you please also put files location in above shell scripts and it should generate a separate directory that contain merge files.

[shenwei356]

Here's an example of redirecting content to file in a directory, dir/file, folder dir will be automatically created if not existed, you need create folder first by mkdir -p dir if it does not exit.

echo hello > dir/file

Answer 2

[ole.tange]

GNU Parallel solutions:

parallel cat {1} {2} '>' {=1 s/_mapped// =} ::: *_mapped.fq :::+ *_unmapped.fq

parallel --xapply cat {1} {2} '>' {=1 s/_mapped// =} ::: *_mapped.fq ::: *_unmapped.fq

parallel cat {} {= s/_unmapped/_mapped/ =} '>' {= s/_unmapped// =} ::: *_unmapped.fq

Read more about GNU Parallel on Gnu Parallel - Parallelize Serial Command Line Programs Without Changing Them

[shenwei356]

thanks for the examples of editing input data in GNU parallel. i did not know input data can be edited by the embeded perl expression, not just the common replacement strings. so i added a RS for removing suffix as shown in the rush examle

[Bioinfonext]

thanks, I never used GNU parallel command. please share link to download to Cent OS platform and after download can I use any of these three command.

[shenwei356]

Please just google GNU parallel or install via yum.

[Bioinfonext]

If I google GNU parallel, it will take me to this page:

https://www.gnu.org/software/parallel/

Which One should I DOWNLOAD for Cent OS platform?

[ole.tange]

See https://www.gnu.org/software/parallel/ under "Community maintained packages".

Or use'the 10 second installation:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

See more on: Gnu Parallel - Parallelize Serial Command Line Programs Without Changing Them

[shenwei356]

Forget GNU parallel if you think it hard to install, use rush instead. rush provides executable binary files for Linux/OS X/Windows. Just download the binary files, decompress and immediately run

Much easier in rush v0.1.8+:

ls *_mapped.fq | rush -v p='{.^_mapped}'  'cat {} {p}_unmapped.fq > outdir/{p}.fq '

[Bioinfonext]

Thank you very much

can you please also share a copy linux command: to coy a directory content into a desired directory.

I use this command: cp -r dir1 dir2

but it create a new directory (dir1) within dir2. I need to only copy contents of dir1 into dir2

[WouterDeCoster]

Try cp dir1/* dir2

[shenwei356]

I think you need read some shell tutorial, or you're going to be frequently stuck by these simple operations.

[Bioinfonext]

yes...surely I will go through the shell tutorial.