Problem Converting BAM to SAM on to Amazon-S3 Using Samtools

Question

[gundalav]

In my Amazon EC2 instance I have the following:

ubuntu@ip-333-31-16-230:~$ df -h
Filesystem       Size  Used Avail Use% Mounted on
udev              79G     0   79G   0% /dev
tmpfs             16G  8.8M   16G   1% /run
/dev/xvda1       985G  136G  809G  15% /
tmpfs             79G     0   79G   0% /dev/shm
tmpfs            5.0M     0  5.0M   0% /run/lock
tmpfs             79G     0   79G   0% /sys/fs/cgroup
tmpfs             16G     0   16G   0% /run/user/1000
bar-something  1.0P     0  1.0P   0% /home/ubuntu/bar-something-data
bar-foolishbrat  1.0P     0  1.0P   0% /home/ubuntu/bar-foolisbrat-data

The last two file system bar-bi and bar-foolishbrat are Amazon S3 bucket, mounted with the following command:

/home/ubuntu/go/bin/goofys bar-foolishbrat bar-foolishbrat-data
/home/ubuntu/go/bin/goofys bar-somehting bar-something-data
# Here we use Goofys. It can be achieved also with S3FS.

Now I have no problem doing this, in my instance

samtools view -h ~/bar-something-data/myfile.bam > test.sam

That is writing the SAM file in my instance file system. But when I do this, namely writing to the mounted S3:

samtools view -h ~/bar-something-data/myfile.bam > ~/bar-something-data/test.sam

It gave the following error:

samtools: writing to standard output failed: Invalid argument
samtools: error closing standard output: -1

How can I resolve this issue? Is there any other tool for converting BAM to SAM other than samtools?

[GenoMax]

Have you tried to use some other means of mounting the file system rather than goofys?

[github]

goofys author here, this shouldn't happen, could you file a bug at https://github.com/kahing/goofys/issues with the debug logs?

[github]

fyi this is fixed: https://github.com/kahing/goofys/issues/154

Answer 1

[Devon Ryan]

I've never seen documentation for it, but as of version 1.3 samtools can directly interact with S3. Have a look at this post from one of the developers for an example. I would suggest posting to the samtools-help mailing list or posting an issue on github for further help regarding this, since it's mostly only the developer team that seems to be using this feature (at the moment).

[John Marshall]

For help using this, please post to the samtools-help mailing list. Filing a GitHub issue is more for when you've run into an actual bug, and is less useful for usage problems like this because the conversation will be seen by fewer of your fellow samtools users.