How to run MD5 check for many fastq files in subdirectories?

Question

[anikcropscience]

Hello,

I have received Illumina sequencing reads for 100 samples. I have 8 R1.fastq.gz and 8 R2.fastq.gz files for each sample in each subfolder. I want to run a MD5 check for all the fastq files in each subfolder.

My folder structure looks like:

/mydata/sequencing/clean
/mydata/sequencing/clean/Sample1
/mydata/sequencing/clean/Sample2
......
/mydata/sequencing/clean/Sample100

I am using the following code being on the /mydata/sequencing/clean directory.

find . -type f -exec md5sum-lite {} \;

This code is printing the md5 result on Terminal.

How do I get these results printed to a .txt file? Besides, is there a more elegant way to do this MD5 check? Thank you.

[cpad0112]

There are three ways to do this depending on destination location and destination file name:

1. you want to store each file's md5sum where each file is located (Remove dry-run if you are okay with dummy run output) :

   $ find . -type f -name "*.pdf" | parallel --dry-run md5sum {} ">" {.}.md5sum

2. you want to store each file's md5sum (one md5sum file for one file) at a single location (current directory in the example below) irrespective of file location and number of md5sum files are equal to number of files:

   $ find . -type f -name "*.pdf" | parallel --plus --dry-run md5sum {} ">" {/.}.md5sum

1. You want to store all the md5sums in a single file in current directory and all MD5sums are written to a single file all.md5 in current directory:

   $ find . -type f -name "*.pdf" | parallel --plus md5sum {} > all.md5

[anikcropscience]

Thanks a lot for the helpful comment.

Answer 1

[sklages]

Just re-direct the output:

find . -type f -exec md5sum-lite {} \; | tee -a md5sums.txt

This way ist straight-forward, thus "elegant". :-)

[anikcropscience]

thank you very much. I have another question. I have a .txt file containing all the md5 identifiers (32 digits) that was given by the sequencing company. Is it possible to verify that file containing all the md5 checks for each file?

[sklages]

yes,

md5sum -c my_checksums.txt

The txt file should contain the actual path to the files to be checked.

[anikcropscience]

Great. Thanks. It worked.

Answer 2

[supertech]

$ md5sum *.fastq >  hashList.txt  #make a file with md5 values of each fastq file
$ md5sum -c hashList.txt   # It runs through list in the  hashList.txt and check one by one.

Output will be on the screen but. You could simply redirect the output to a file.

[anikcropscience]

But will it run over all the .fastq files in different subfolders being on the parent directory?

[sklages]

No, it won't run recursive. Your find-approach is working well, no need to search for another solution :-)

[anikcropscience]

Ok cool. Thanks a lot.

[supertech]

sklages answered to this above.

Answer 3

[yhoogstrate]

I would also recommend you to run gzip --test *.gz. Even if the gzip files are succesfully transferred (for which you actually check using md5), it does not guarantee that the files are not corrupt. This also works for BAM files, which are effectively valid gzip files.