Nextflow ERROR : Timeout waiting for connection from pool

Question

[dzisis1986]

Hi it is my first time trying to use nf-core sarek pipeline in nextflow . I want to run only the annotation part with input vcf. I am getting the following error and I have no clue or idea how to solve it Do you know any solution or to what this ERROR is related ?

$ nextflow run nf-core/sarek -r 3.4.0 profile test --input AC1/vcf_single_AC1.csv --outdir testResutsAC1_new --step annotate --tools snpeff

ERROR ~ Error executing process > 'NFCORE_SAREK:SAREK:PREPARE_GENOME:UNZIP_RT'

Caused by:
  Timeout waiting for connection from pool


 -- Check '.nextflow.log' file for details

[Pierre Lindenbaum]

cross posted: https://stackoverflow.com/questions/78202194

[dzisis1986]

Yes sorry for that but they didnt reply to that post

[Pierre Lindenbaum]

could be a temporary problem with AWS: https://github.com/nextflow-io/nextflow/issues/856

[GenoMax]

Have you looked in .nexfflow.log files for additional details as the suggestion above notes?

[dzisis1986]

Yes this is the part with the error :

Mar-21 16:21:57.696 [S3TransferManager-49] WARN  c.a.s.s.i.S3AbortableInputStream - Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.
Mar-21 16:21:57.704 [Actor Thread 442] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=NFCORE_SAREK:SAREK:PREPARE_GENOME:UNZIP_RT; work-dir=null
  error [com.amazonaws.SdkClientException]: Unable to execute HTTP request: Timeout waiting for connection from pool
Mar-21 16:21:57.705 [S3TransferManager-39] WARN  c.a.s.s.i.S3AbortableInputStream - Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.
Mar-21 16:21:57.718 [S3TransferManager-31] WARN  c.a.s.s.i.S3AbortableInputStream - Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.
Mar-21 16:21:57.724 [Actor Thread 442] ERROR nextflow.processor.TaskProcessor - Error executing process > 'NFCORE_SAREK:SAREK:PREPARE_GENOME:UNZIP_RT'

Caused by:
  Timeout waiting for connection from pool

com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1165)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)

[GenoMax]

Looks to be similar to the thread @pierre posted. Can you try the option indicated here: https://github.com/nextflow-io/nextflow/issues/856#issuecomment-1933259446

[dzisis1986]

i will try it and let you know it can be something like that

[dzisis1986]

It doesn't make sense how to change the threads and in my case it is not working! What kind of problem is that ? is it related to Amazon to the memory of my computer, to a number of threads, or to the internet connection? It is completely strange to me and I cant figure out how to solve it !

[GenoMax]

As best as I can tell it is a combination of things. Did you try setting export NXF_ENABLE_VIRTUAL_THREADS="true" in your shell? You also need to upgrade to Java 21.

nextflow has a slack channel where all their developers are active. You can join and post this problem there and/or link to this thread to see if they have a solution: https://nextflow.slack.com/signup#/domain-signup

[dzisis1986]

Yes, I tried the export but itdoesn't work and I have Java 21 on my Mac! I think it's a combination of problems and whatever I tried doesn't work!