Read bam/cram file with IGV from aws s3
2
4
Entering edit mode
2.9 years ago
quentin54520 ▴ 120

Hi all,

We store our alignment files on aws s3. I would like to be able to open them with IGV without needing to download them completely, but I can't find an optimal solution.

If I get a pre-signed url it works but it's not convenient.

I try to follow this but I can't do it at all. I think the problem comes from the configuration file at the end, I don't know exactly what is needed or not. Knowing that we do not have a connection by google, I would just like to use our aws identifiers. When i click on login on igv and add my longin and password i receive a "succes" answer; but then nothing happen on igv, the menu "amazon --> Load from s3 bucket is still inaccessible.

i use this json (i replace the XXXX by the corresponding value):

   {
   "apiKey": "",

   "project_id": "igv",

   "auth_provider": "Amazon",
   "aws_region": "us-east-1",
   "scope": "email",
   "redirect_uris": [
     "http://localhost:60151/oauthCallback"
   ],

   "client_id": "XXXXXX",
   "client_secret": "XXXXX",
   "authorization_endpoint": "XXXXXX",
   "token_endpoint": "XXXX",
   "aws_cognito_pool_id": "XXXX",
   "aws_cognito_fed_pool_id": "",
   "aws_cognito_role_arn": "XXXXX"
}
s3 IGV aws amazon • 4.9k views
ADD COMMENT •
1
Entering edit mode

It's been on my roadmap to upgrade our current solution to using federated access. For now, we have all the alignments stored in a public S3 bucket but restrict access to certain IP addresses and VPC endpoints. It works for both IGV (end-users need to be connected via VPN) and our internal genome browser.

To limit access on S3 buckets: https://aws.amazon.com/premiumsupport/knowledge-center/block-s3-traffic-vpc-ip/

ADD REPLY •
0
Entering edit mode

May want to ask this on IGV support forum: https://groups.google.com/g/igv-help

ADD REPLY •
2
Entering edit mode
2.4 years ago
quentin54520 ▴ 120

if people come across this post here is some new information:

The last version of igv (2.13.0) add a native support for s3. You just have to get your credentials (programmatic acces) acces key and secret Key, put your credentials files in the good directory (usually it's ~/.aws/credentials ) and now you can click on the Amazon tab in IGV to load your files.
ADD COMMENT •
3
Entering edit mode
2.9 years ago

Author of IGV's AWS S3+Cognito support here... can you elaborate on what's failing for you (share some error messages from IGV's console when running it from cmdline)?

The success message is a good sign...Did you follow the steps in:

https://umccr.org/blog/igv-amazon-backend-setup/

Make sure you review the S3 policies (paste it over here), that's a common error: working auth yet insufficient S3 perms.

The JSON you shared looks incomplete, here's our working example for our org:

https://github.com/umccr/infrastructure/tree/master/cdk/apps/igv/config/backend/dev

Also, there's the frontend docs too over here:

https://umccr.org/blog/igv-amazon-frontend-setup/

Hope that helps!
ADD COMMENT •
0
Entering edit mode

Thank you for your reply,

Yes i follow all the steps of this link.

For the json i don't know what to put on "aws_cognito_fed_pool_id" or "auth_provider_x509_cert_url" is i don't want to used federated acces, but just from the aws cognito user.

The error of IGV after click on "amazon --> Login, add the login and password, i come back to igv and i get :

INFO [2022-01-20T17:50:03,629] [CommandListener.java:186]  GET /oauthCallback?code=30eceb29-f186-4837-b19f-791ee492c6a1&state=94093997-bbaa-4a01-b4b1-93b9dabcde18 HTTP/1.1
ERROR [2022-01-20T17:50:03,673] [OAuthProvider.java:242]  java.net.UnknownHostException: igv-laporte-team.igv-laporte-team.auth.us-east-1.amazoncognito.com
INFO [2022-01-20T17:50:04,280] [CommandListener.java:186]  GET /favicon.ico HTTP/1.1

The S3 iam role is :

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "IGVCognitoAuthedUsersProd",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::mybucket"
            ]
        },
        {
            "Sid": "IGVListBuckets",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "mobileanalytics:PutEvents",
                "cognito-sync:*",
                "cognito-identity:*"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}
ADD REPLY •
0
Entering edit mode

ERROR [2022-01-20T17:50:03,673] [OAuthProvider.java:242] java.net.UnknownHostException: igv-laporte-team.igv-laporte-team.auth.us-east-1.amazoncognito.com

This hostname repetition, igv-laporte-team.igv-laporte-team. ... should be just one of those, can you double check the configs for that one?

Also make sure you download the .json.gz oauth-config file I pointed above and compare it with yours.

ADD REPLY •
0
Entering edit mode

i managed this error but now i'm able to see my list of bucket but when i choose a cram file and click on load i get an error :

Unexpected error: null (Service: S3, Status Code: 403, Request ID: null, Extended Request ID: s27xAGqHihSGR+CZw+v5gQO9bO5r2XflepSrLWflIr4UBODw7ktoUYmxMBVWeiJ9aSGx+PZYV84=). See igv.log for more details

The complete log of igv :

INFO [2022-01-21T17:56:16,381] [MessageUtils.java:76]  <html>Unexpected error: null (Service: S3, Status Code: 403, Request ID: null, Extended Request ID: s27xAGqHihSGR+CZw+v5gQO9bO5r2XflepSrLWflIr4UBODw7ktoUYmxMBVWeiJ9aSGx+PZYV84=).<br>See igv.log for more details
ERROR [2022-01-21T17:56:16,384] [LongRunningTask.java:75]  Exception running task
software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 403, Request ID: null, Extended Request ID: s27xAGqHihSGR+CZw+v5gQO9bO5r2XflepSrLWflIr4UBODw7ktoUYmxMBVWeiJ9aSGx+PZYV84=)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156) ~[aws-xml-protocol-2.15.9.jar:?]
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:106) ~[aws-xml-protocol-2.15.9.jar:?]
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:84) ~[aws-xml-protocol-2.15.9.jar:?]
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:42) ~[aws-xml-protocol-2.15.9.jar:?]
    at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:94) ~[aws-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$5(BaseClientHandler.java:229) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:133) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:159) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:112) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:167) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:94) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[sdk-core-2.15.9.jar:?]
    at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) ~[aws-core-2.15.9.jar:?]
    at software.amazon.awssdk.services.s3.DefaultS3Client.headObject(DefaultS3Client.java:4875) ~[s3-2.15.9.jar:?]
    at org.broad.igv.util.AmazonUtils.getObjectMetadata(AmazonUtils.java:211) ~[igv.jar:?]
    at org.broad.igv.util.AmazonUtils.isObjectAccessible(AmazonUtils.java:252) ~[igv.jar:?]
    at org.broad.igv.aws.S3LoadDialog.lambda$loadButtonActionPerformed$0(S3LoadDialog.java:106) ~[igv.jar:?]
    at org.broad.igv.util.LongRunningTask.call(LongRunningTask.java:72) [igv.jar:?]
    at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
    at java.lang.Thread.run(Unknown Source) [?:?]
ADD REPLY •
0
Entering edit mode

Oh this time the issue was the IAM, now it's work ! Thanks you so much

The only point that i found strange is that after the first login, if i restart my computer i can log in without enter my password. It's normal ? I made a mistake in the configuration ?

ADD REPLY •
0
Entering edit mode

Great to hear!

What was the problem with IAM? Can you describe how you solved that a bit more?

If you haven't closed IGV or if your computer's OS session management restores processes as they were, yes, the temporal access creds are stored within IGV's process memory.

ADD REPLY •
0
Entering edit mode

Thanks for you assistance Roman, as always. Just a reminder that you can always open an issue at https://github.com/igvteam/igv/issues. I personally don't have time to scan biostars regularly.

ADD REPLY •
0
Entering edit mode

First I had a problem of misunderstanding what is "aws_cognito_fed_pool_id" now it's fixed. The only difference with your config file is that I left "auth_provider_x509_cert_url" empty.

For the iam role i forgot the second line in the the field "ressource" of the sid "IGVCognitoAuthedUsersProd" .

Another point that has nothing to do with the error: I left the configuration file on my computer rather than putting it in a public bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "IGVCognitoAuthedUsersProd",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::mybucket",
                "arn:aws:s3:::mybucket/*"
            ]
        },
        {
            "Sid": "IGVListBuckets",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "mobileanalytics:PutEvents",
                "cognito-sync:*",
                "cognito-identity:*"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}
ADD REPLY •

Login before adding your answer.

Similar Posts
Loading Similar Posts
Traffic: 2421 users visited in the last hour
Content Search
Users
Tags
Badges
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6