Using tabix remotely against S3

Question

[andrewl]

Hi everyone,

I have tabix working well locally but want to store my genomes on an S3 bucket (index files will likely stay local). I am having trouble getting this to work as advertised.

wget https://s3-eu-west-1.amazonaws.com/*/WGC069899D-hg38-sorted.vcf.gz

Resolving s3-eu-west-1.amazonaws.com... ...

Connecting to s3-eu-west-1.amazonaws.com|...|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 443976024 (423M) [application/x-gzip]

Saving to: 'WGC069899D-hg38-sorted.vcf.gz.1'

So no (permission) problems with accessing the file from this bucket.

But when I try tabix I get:

tabix https://s3-eu-west-1.amazonaws.com/*/WGC069899D-hg38-sorted.vcf.gz chr1:1000-2000

[E::hts_open_format] fail to open file 'https://s3-eu-west-1.amazonaws.com/*/WGC069899D-hg38-sorted.vcf.gz'

Could not read https://s3-eu-west-1.amazonaws.com/*/WGC069899D-hg38-sorted.vcf.gz

When I do the same tbix on the file locally, it works fine?

What am I missing?

Thanks.

[Pierre Lindenbaum]

stupid questions:

• is the *.tbi file present on the amazon side ?
• are you working behind a proxy ?
• does amazon supports Byte-Range request ? https://en.wikipedia.org/wiki/Byte_serving

[Devon Ryan]

What happens if you instead tabix s3://*/WGC069899D-hg38-sorted.vcf.gz chr1:1000-2000, where * is the bucket on S3? At least for bam/cram files that sort of thing works.

Answer 1

[Santosh Anand]

The problem might be the https: protocol. See this https://github.com/samtools/htslib/issues/139

TL; DR: Check that your htslib is current, and it has been installed with libcurl and libcrypto libraries. See requirements at https://github.com/samtools/htslib/blob/develop/INSTALL

HTSlib uses the following external libraries. Building requires both the library itself, and include files needed to compile code that uses functions from the library. Note that some Linux distributions put include files in a development ('-dev' or '-devel') package separate from the main library.

libz       (required)
libbz2     (required, unless configured with --disable-bz2)
liblzma    (required, unless configured with --disable-lzma)
libcurl    (optional, but strongly recommended)
libcrypto  (optional for Amazon S3 support; not needed on MacOS)
  

[lh3]

^This. You need to build tabix with libcurl+openssl.

[shubhamnath5]

hello. Can you help me with the same issue. I am new to this environment of tabix and s3

[shubhamnath5]

hello. Can you help me with the same issue. I am new to this environment of tabix and s3