Samtools (v 1.5) index: failed to create index after sorting by name

Question

[GRT]

Hello

I was wondering if anybody could explain the following:

Samtools 1.5 on a SGE setup (modules available are Samtools 0.1.19, 1.2 or 1.5), Calling Samtools from within a shell script.

If I run this sort command:

samtools sort -n contig_XXX_mapped.latest.all.bam -o contig_XXX_mapped.sorted.bam

Then I get a bam file contig_XXX_mapped.sorted.bam in the relevant folder

If I then run:

samtools index contig_XXX_mapped.sorted.bam

We get the error: samtools index: failed to create index for "contig_XXX_mapped.sorted.bam": No such file or directory - as stated that file does exist.

Repeating the sort command without '-n' works just fine - an output file of the same name is created and is indexed.

I cannot find an explanation for this behaviour and I was wondering if anybody could explain it to me?

Many thanks

[prince26121991]

I tried to reproduce your problem with v1.5, but I didn't get the error what you are having, what I am getting is genuine issue, because I have sorted it on the basis of name. Therefore I think samtools is not able to access your sorted file due to permission. As you are in a cluster environment, just make sure samtools is able to access your input file.

scbb@scbb-sm:~/Downloads/Compressed/samtools-1.5$ ./samtools sort -n Col0_C1.100k.sorted.bam -o contig_XXX_mapped.sorted.bam
scbb@scbb-sm:~/Downloads/Compressed/samtools-1.5$ ./samtools index contig_XXX_mapped.sorted.bam
[E::hts_idx_push] Unsorted positions on sequence #1: 50540 followed by 50496
samtools index: failed to create index for "contig_XXX_mapped.sorted.bam"

[GRT]

Hi - thanks for the reply, Bit of digging further How to specify the sort based on name in samtools sort? - looks like samtools reports file not found when, in fact, it just cannot index a name sorted file as described below by ATpoint.

[prince26121991]

But following solution doesn't give the answer of your question. Because I am also getting same error when sorting by name as described in"How to specify the sort based on name in samtools sort? " . But your error is totally different then what is expected. I am still curious why you are getting that error.

[ATpoint]

It has nothing to do with samtools:

$samtools --version && samtools index nsort.bam 
samtools 1.5
Using htslib 1.5
Copyright (C) 2017 Genome Research Ltd.
[E::hts_idx_push] Unsorted positions on sequence #1: 4807943 followed by 4645631
samtools index: failed to create index for "nsort.bam"

This is most likely due to the wrapper script that is causing this behaviour, somewhere parsing a wrong file name or something similar.

Answer 1

[ATpoint]

Indexing requires coordinate-sorted files. Name-sorted does not work. This might not explain why this file cannot be found but anyway, what you aim to do is pointless. Without knowing more about the code you used to wrap this samtools command in it is difficult to find out why the file cannot be found.

[GRT]

Hi - thanks for the reply. As above - it looks as though samtools is reporting 'file not found' when the file is sorted by name. Possible this is referring to the expected output reference file? Probably due to the fact that you have pointed out that name-sorted will not index. Thank you

[mgdias.jose]

I have the exact same issue. I have alignments generate with mummer4, then converted SAM to BAM view samtools -bT. Finally, the sorting by query name led to the same issue when I have tried to do it with bam files sorted by query name. (Picard SamSort) Just weird the No such file or directory message

[ATpoint]

Your post lacks any details on what exactly you did and what went wrong. Please critically review your code and if you still need help, open a new question.

Answer 2

[sagitaninta]

Hi, I got the same problem previously, the samtools index cannot find my file although it is there. Apparently it is because I made a stupid mistake in defining the output my mapped bam through samtools.

Here is my stupid code that made the samtools index cannot find my files:

bwa mem -t 8 $REFDIR <(gunzip -c $INPUT1) <(gunzip -c $INPUT2) | samtools view -Shb - > $OUTPUT | samtools sort -m 2G -@ 4 - -o $OUTPUT
samtools index $OUTPUT

I do not know what happened between the pipes, but after I corrected it into ...

bwa mem -t 8 $REFDIR <(gunzip -c $INPUT1) <(gunzip -c $INPUT2) | samtools view -Shu - | samtools sort -m 2G -@ 4 - -o $OUTPUT
samtools index $OUTPUT

.. or ...

bwa mem -t 8 $REFDIR <(gunzip -c $INPUT1) <(gunzip -c $INPUT2) | samtools view -Shb > $OUTPUT 
samtools sort -m 2G -@ 4 $OUTPUT -o $OUTPUT2
samtools index $OUTPUT2

samtools index got them just fine.

So probably the problem is in the bam file, it is not readable, or permission issue. In my case, I just re-mapped it again.

[ATpoint]

It is normal and expected what you see because with > $OUTPUT in the first command you are sending the stream to disk instead of sending it into samtools sort.

The second command is fine as there you do send to stream to the sort command and in the third one you first save it do disk and then load it back to sort. Options 2) and 3) are fine, 2) is more efficient.

By the way bwa will happily accept gzipped files, no need to manualyl decompress them. Just feed in the compressed files. Also, if you upgrade to the newer samtools versions you do not need to use the view command as sort can read SAM format directly from bwa.