Nextflow crash on AWS Batch with transition error

Question

[hkarakurt]

Hello everyone, I am having a problem with AWS Batch.

executor >  awsbatch (57)
[d6/1f982d] process > BWA_ALIGNMENT (Alignment on R187-POOL-25_S25_L001)          [100%] 6 of 6, failed: 1
[c6/f78a3c] process > FASTQC (FASTQC on R187-POOL-25_S25_L001)                    [100%] 24 of 24
[-        ] process > MULTIQC                                                     -
[-        ] process > MARKDUPLICATES (MarkDuplicates on R187-POOL-01_S1_L001.bam) -
[-        ] process > BASERECALIBRATOR                                            -
[-        ] process > APPLYBQSR                                                   -
[-        ] process > POOL_SPLITTER                                               -
[-        ] process > VARIANT_CALLING                                             -
[-        ] process > VCF_ANNOTATION                                              -
[-        ] process > VCF2TSV                                                     -
ERROR ~ Error executing process > 'BWA_ALIGNMENT (Alignment on R187-POOL-16_S16_L001)'

Caused by:
  Essential container in task exited - CannotInspectContainerError: Could not transition to inspecting; timed out after waiting 30s

Command executed:

  bwa mem -M -R '@RG\tID:Sample\tLB:lib\tSM:Sample\tPL:illumina\tCN:Sample' -t 12 hg19_v0_Homo_sapiens_assembly19.fasta R187-POOL-16_S16_L001_R1_001.fastq.gz R187-POOL-16_S16_L001_R2_001.fastq.gz  | samtools sort -@ 12 -o R187-POOL-16_S16_L001.bam -

Command exit status:
  0

Command output:
  (empty)

Command error:
  [M::bwa_idx_load_from_disk] read 0 ALT contigs
  [M::process] read 659908 sequences (74763087 bp)...
  [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (57, 86100, 105, 49)
  [M::mem_pestat] analyzing insert size distribution for orientation FF...
  [M::mem_pestat] (25, 50, 75) percentile: (101, 214, 492)
  [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 1274)
  [M::mem_pestat] mean and std.dev: (287.94, 254.84)
  [M::mem_pestat] low and high boundaries for proper pairs: (1, 1665)
  [M::mem_pestat] analyzing insert size distribution for orientation FR...
  [M::mem_pestat] (25, 50, 75) percentile: (63, 97, 144)
  [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 306)
  [M::mem_pestat] mean and std.dev: (108.09, 57.59)
  [M::mem_pestat] low and high boundaries for proper pairs: (1, 387)
  [M::mem_pestat] analyzing insert size distribution for orientation RF...
  [M::mem_pestat] (25, 50, 75) percentile: (86, 184, 859)
  [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2405)
  [M::mem_pestat] mean and std.dev: (288.90, 419.84)
  [M::mem_pestat] low and high boundaries for proper pairs: (1, 3178)
  [M::mem_pestat] analyzing insert size distribution for orientation RR...
  [M::mem_pestat] (25, 50, 75) percentile: (80, 206, 523)
  [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 1409)
  [M::mem_pestat] mean and std.dev: (266.86, 277.91)
  [M::mem_pestat] low and high boundaries for proper pairs: (1, 1852)
  [M::mem_pestat] skip orientation FF
  [M::mem_pestat] skip orientation RF
  [M::mem_pestat] skip orientation RR
  [M::mem_process_seqs] Processed 659908 reads in 235.092 CPU sec, 19.759 real sec
  [main] Version: 0.7.17-r1188
  [main] CMD: bwa mem -M -R @RG\tID:Sample\tLB:lib\tSM:Sample\tPL:illumina\tCN:Sample -t 12 hg19_v0_Homo_sapiens_assembly19.fasta R187-POOL-16_S16_L001_R1_001.fastq.gz R187-POOL-16_S16_L001_R2_001.fastq.gz
  [main] Real time: 25.762 sec; CPU: 240.056 sec
  [bam_sort_core] merging from 0 files and 12 in-memory blocks...

Work dir:
  s3://gene2var-nextflow/test_folder/78/7180148c2cb524b4379b8c0ba18063

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details
[AWS BATCH] Waiting jobs reaper to complete (20 jobs to be terminated)

Sometimes I am having the same issue in another steps but I do not know how to solve it. I am using an AWS Linux ECS-Optimized AMI with 8 GB root and 142 GB EBS volume. What might be the problem here?

My config:

params {
    reads = "s3://R187/*R{1,2}_001.fastq.gz"
    genome_file = "s3:///hg19_v0_Homo_sapiens_assembly19.fasta"
    genome_dict = "s3:///hg19_v0_Homo_sapiens_assembly19.dict"
    genome_file_idx = "s3:///hg19_v0_Homo_sapiens_assembly19.fasta.fai"
    genome_loc = "s3://"
    known_sites = "s3://references/dbsnp/hg19_v0_Homo_sapiens_assembly19.dbsnp138.vcf.gz"
    known_sites_idx = "s3://references/dbsnp/hg19_v0_Homo_sapiens_assembly19.dbsnp138.vcf.gz.tbi"
    outdir = "s3://test_folder/R187_results"
    pool_info = "s3://BED_Files/R187_pool_info.txt"
    bed_location = "s3://BED_Files/R187_bed_files/"
    vep_cache = "s3://vep_data"
    vep_plugin = "s3://vep_data/Plugins"
    cpus = "12"
}

profile {
aws {
    accessKey = '***********'
    secretKey = '******************************'
    docker.enabled = true
    region = 'eu-west-3'
    }
}


aws.batch.maxParallelTransfers = 5

process.executor = 'awsbatch'

process.container = 'aligner:latest'

process.container = 'annotation:latest'

process.container = 'variant_caller:latest'

process.queue = 'batch_queue'

aws.batch.cliPath = '/home/ec2-user/miniconda/bin/aws'

workDir = 's3://test_folder/'

Why I am having this error?

[colindaven]

I'd post this error here: https://github.com/nextflow-io/nextflow/discussions

Answer 1

[barslmn]

BWA output looks normal. The error is at this part which is an ECS related issue:

  Essential container in task exited - CannotInspectContainerError: Could not transition to inspecting; timed out after waiting 30s

Searching for it returns two relevant discussions:

https://github.com/aws/amazon-ecs-agent/issues/2312

https://github.com/nextflow-io/nextflow/issues/3091

The one on Nextflow repo has the solution of updating the AMI.