Interpretation Of Consensus Sequence From Samtools Pileup

Question

[Ian]

I have generated a consensus sequence using 'pileup' then 'pileup2fq' from samtools. Can anyone tell me exactly what determines whether the resulting sequence is in UPPER or lower case?

An example of the fastq is:

@header
GTTAAGATGAAACATTTACAGGATTTGATTGACGAACCTGATGAtttttcacaacccaat ccatCTtagactagaaaggtaTTTACGGTTGCTaaacattgcgttatgtttaaGACCTCA TGCCAATAGACTGTTTGAATTTTATGAactgtctcctttgggaaacttgttaagtcgtga aastnnnnnnnnnnnnnnncaagggtacttggtcatcagatctaccgcaaaagctCAAGG
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~r oZMH!!!!!!!!!!!!!!!KZo~~Z~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Thanks.

Answer 1

[Ian]

Thanks for viewing, but i found my answer here:

samtools.pl pileup2fq -D100 > var-X.fq

File var-X.fq is in the multi-line FASTQ format. Bases in lowercase are filtered out due to repeats, being close to indels or insufficient/excessive read depth. The consensus file is essential to the estimate of mutation rate. The pileup2fq command applies fewer filters than varFilter and may not give identical results.

[Neilfws]

In this case, I guess you can mark your own answer as the best :-)

[Ian]

I wish i hadn't asked it now....

[Doctoroots]

so just to make sure, the lower case letters were either covered but filtered out due to the aforementioned reasons or they were just not covered in any of the reads?

[Ian]

From my experience, if they are not at all covered then you do not get any bases in the fastq file.

[Love]

I run the command but the lower case letters are not filtered at all. Why?

[jackuser1979]

Hi Ian. Just curious. After coverting to FASTQ format, how you end up in estimation of mutation rate (which software/tools do you used)?

Answer 2

[Pablo Marin-Garcia]

The lower case letter are what is called 'soft masking' (bases in low complexity regions like repeats etc.). I don't know if it would be your case but some people provide the reference genome soft masked to the aligners in order to avoid alignments in this regions, but nowadays this is saw as a not good practice and I think that aligners like BWA does not filter out soft masked regions following this biostars answer and similar posts

[Casey Bergman]

Ian's answer below is correct, not this one. Lowercase letters in samtools output does not mean the same thing as typical soft-masking: http://sourceforge.net/apps/mediawiki/samtools/index.php?title=SAM_protocol#Basic_Protocol_3:_Variant_Calling_with_SAMtools

[Pablo Marin-Garcia]

+1 @Casey: You are right Casey, probably I did not express myself with property. When I was talking about soft-masking I was meaning marking bases to be filtered out. Although the sof-masking rules can be slightly different the concept is the same, as the samtools manual says: "Bases in lowercase are filtered out due to repeats, being close to indels or insufficient/excessive read depth"