Difference between samtools "-f 9" and "-f 11"

Question

[Dylan]

So ran into an interesting outcome from samtools filtering. I'm attempting to select reads in which one member of the pair is mapped, but the other is not. After searching for the best filtering for that, most people seemed to indicate -f 11 was the correct choice for this. However, that includes "properly paired" which seems like something I don't want. Additionally when I used -f 11 I get nothing coming out. I decided to try -f 9 which removes the proper pairing component and that seemed to work for my needs.

Can anyone explain why the posts I've seen in the past suggest -f 11 and whether or not this has an internal contradiction which yields no reads?

Answer 1

[Pierre Lindenbaum]

9: read must be paired AND mate must be unmapped

11: read must be paired AND mapped in proper pair AND mate must be unmapped (which is weird, if it's in proper pair, read and its mate should be both mapped)

I'm attempting to select reads in which one member of the pair is mapped, but the other is not.

I think you want:

samtools view --expr 'flag.paired && ((flag.unmap && !flag.munmap) || (!flag.unmap && flag.munmap))'  in.bam

[Dylan]

(which is weird, if it's in proper pair, read and its mate should be both mapped)

So that was my thought as well, I think the code you listed essentially mirrors doing -f 9 and -f 5 together

[Istvan Albert]

It is very easy to make nonsensical flags (unfortunately a flaw of the SAM spec itself) ... not to mention the awkward constructs: a primary alignment is an alignment that is not secondary, not supplementary and not unmapped ...