Question

CombineGVCFs skips a chromosome

0

Entering edit mode

13 months ago

Matteo ▴ 10

Hi!

I am having issues for the first time with CombineGVCFs. Specifically, it outputs a combined gvcf without chromosome 8 (SUPER_8) even though this is present in the individual gvcfs that I input in the command. There is no error in the log file, the engine just shuts down after chromosome 7, just before chromosome 8. Any idea about what may be going on? Any suggestion will be much appreciated. Thanks in advance!

Matteo

REQUIRED for all errors and issues: a) GATK version used: gatk-4.2.6.1

b) Exact command used:

java -Xmx128g -jar gatk-4.2.6.1/gatk-package-4.2.6.1-local.jar CombineGVCFs \

        -R /home/vonholdt/VONHOLDT/reference_genomes/Pogoniulus_pusillus/bPogPus1_combined_assembly.fasta \

        -V 230188_pre_bqsr.g.vcf.gz \

        -V 990025_pre_bqsr.g.vcf.gz \

        -V A241_pre_bqsr.g.vcf.gz \

        -V AR69209_pre_bqsr.g.vcf.gz \

        -O ../3_Joint_Genotyping/coryphaea1_post_bqsr_gatk.g.vcf.gz

c) Entire program log:

04:39:23.337 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/projects/VONHOLDT/BIN/gatk-4.2.6.1/gatk-package-4.2.6.1-local.jar!/com/intel/gkl/native/libgkl_compression.so

04:39:23.669 INFO  CombineGVCFs - ------------------------------------------------------------

04:39:23.670 INFO  CombineGVCFs - The Genome Analysis Toolkit (GATK) v4.2.6.1

04:39:23.670 INFO  CombineGVCFs - For support and documentation go to https://software.broadinstitute.org/gatk/

04:39:23.670 INFO  CombineGVCFs - Executing as ms0553@della-r2c2n9 on Linux v4.18.0-477.27.1.el8_8.x86_64 amd64

04:39:23.670 INFO  CombineGVCFs - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_382-b05

04:39:23.670 INFO  CombineGVCFs - Start Date/Time: November 1, 2023 4:39:23 AM EDT

04:39:23.670 INFO  CombineGVCFs - ------------------------------------------------------------

04:39:23.670 INFO  CombineGVCFs - ------------------------------------------------------------

04:39:23.671 INFO  CombineGVCFs - HTSJDK Version: 2.24.1

04:39:23.671 INFO  CombineGVCFs - Picard Version: 2.27.1

04:39:23.671 INFO  CombineGVCFs - Built for Spark Version: 2.4.5

04:39:23.671 INFO  CombineGVCFs - HTSJDK Defaults.COMPRESSION_LEVEL : 2

04:39:23.671 INFO  CombineGVCFs - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false

04:39:23.671 INFO  CombineGVCFs - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true

04:39:23.671 INFO  CombineGVCFs - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false

04:39:23.671 INFO  CombineGVCFs - Deflater: IntelDeflater

04:39:23.671 INFO  CombineGVCFs - Inflater: IntelInflater

04:39:23.671 INFO  CombineGVCFs - GCS max retries/reopens: 20

04:39:23.671 INFO  CombineGVCFs - Requester pays: disabled

04:39:23.671 INFO  CombineGVCFs - Initializing engine

04:39:24.021 INFO  FeatureManager - Using codec VCFCodec to read file file:///scratch/gpfs/ms0553/matteo/Pogoniulus_study/snp_calling/4_post_bqsr/2_Combined_VCFs/230188_pre_bqsr.g.vcf.gz

04:39:24.151 INFO  FeatureManager - Using codec VCFCodec to read file file:///scratch/gpfs/ms0553/matteo/Pogoniulus_study/snp_calling/4_post_bqsr/2_Combined_VCFs/990025_pre_bqsr.g.vcf.gz

04:39:24.247 INFO  FeatureManager - Using codec VCFCodec to read file file:///scratch/gpfs/ms0553/matteo/Pogoniulus_study/snp_calling/4_post_bqsr/2_Combined_VCFs/A241_pre_bqsr.g.vcf.gz

04:39:24.337 INFO  FeatureManager - Using codec VCFCodec to read file file:///scratch/gpfs/ms0553/matteo/Pogoniulus_study/snp_calling/4_post_bqsr/2_Combined_VCFs/AR69209_pre_bqsr.g.vcf.gz

04:39:25.050 INFO  CombineGVCFs - Done initializing engine

04:39:25.128 INFO  ProgressMeter - Starting traversal

04:39:25.128 INFO  ProgressMeter -        Current Locus  Elapsed Minutes    Variants Processed  Variants/Minute

04:39:25.508 WARN  ReferenceConfidenceVariantContextMerger - Detected invalid annotations: When trying to merge variant contexts at location scaffold_261_arrow_ctg1:8487 the annotation MLEAC=[0, 0] was not a numerical value and was ignored

04:39:35.376 INFO  ProgressMeter -     SUPER_26:1247096              0.2               1285000        7708458.3

04:39:45.131 INFO  ProgressMeter -     SUPER_26:2866464              0.3               2800000        8398740.2

04:39:55.131 INFO  ProgressMeter -     SUPER_26:4449520              0.5               4269000        8537146.3

04:40:05.133 INFO  ProgressMeter -     SUPER_26:6022575              0.7               5836000        8752905.9

04:40:15.134 INFO  ProgressMeter -     SUPER_26:7535995              0.8               7394000        8871735.4

04:40:25.138 INFO  ProgressMeter -     SUPER_26:9066749              1.0               8946000        8944509.2

04:40:35.141 INFO  ProgressMeter -    SUPER_26:10634261              1.2              10497000        8995757.9

04:40:45.147 INFO  ProgressMeter -    SUPER_26:12218206              1.3              12044000        9030855.2

04:40:55.151 INFO  ProgressMeter -    SUPER_26:13951281              1.5              13596000        9061684.2

04:41:05.155 INFO  ProgressMeter -    SUPER_26:15701833              1.7              15155000        9090545.6

04:41:15.156 INFO  ProgressMeter -    SUPER_26:17423090              1.8              16709000        9111763.5

04:41:25.158 INFO  ProgressMeter -    SUPER_26:19345502              2.0              18284000        9139791.2

04:41:35.159 INFO  ProgressMeter -      SUPER_17:485882              2.2              19862000        9164961.9

04:41:45.164 INFO  ProgressMeter -     SUPER_17:2540828              2.3              21457000        9193493.1

04:41:55.169 INFO  ProgressMeter -     SUPER_17:4337676              2.5              23023000        9206683.5

04:42:05.170 INFO  ProgressMeter -     SUPER_17:5801941              2.7              24598000        9221829.3

04:42:15.175 INFO  ProgressMeter -     SUPER_17:7256963              2.8              26169000        9233564.8

04:42:25.177 INFO  ProgressMeter -     SUPER_17:8715266              3.0              27740000        9244150.2

04:42:35.180 INFO  ProgressMeter -    SUPER_17:10153900              3.2              29312000        9253888.4

04:42:45.180 INFO  ProgressMeter -    SUPER_17:11628308              3.3              30862000        9256193.4

04:42:55.185 INFO  ProgressMeter -    SUPER_17:13090994              3.5              32414000        9258629.8

04:43:05.189 INFO  ProgressMeter -    SUPER_17:14589591              3.7              33963000        9260068.8

04:43:15.190 INFO  ProgressMeter -    SUPER_17:16214217              3.8              35502000        9258895.4

04:43:25.195 INFO  ProgressMeter -    SUPER_17:17752636              4.0              37040000        9257415.6

04:43:35.199 INFO  ProgressMeter -    SUPER_17:19245135              4.2              38573000        9254928.6

04:43:45.199 INFO  ProgressMeter -    SUPER_17:20929710              4.3              40100000        9251319.8

...

05:39:06.267 INFO  ProgressMeter -     SUPER_7:25969906             59.7             546305000        9153037.6

05:39:16.270 INFO  ProgressMeter -     SUPER_7:27717550             59.9             547922000        9154558.6

05:39:26.273 INFO  ProgressMeter -     SUPER_7:29399186             60.0             549486000        9155188.1

05:39:36.274 INFO  ProgressMeter -     SUPER_7:30952335             60.2             550981000        9154672.8

05:39:46.278 INFO  ProgressMeter -     SUPER_7:32675838             60.4             552553000        9155428.5

05:39:56.279 INFO  ProgressMeter -     SUPER_7:34544117             60.5             554182000        9157129.5

05:40:06.281 INFO  ProgressMeter -     SUPER_7:36497744             60.7             555807000        9158752.7

05:40:16.282 INFO  ProgressMeter -     SUPER_7:38325768             60.9             557352000        9159054.9

05:40:26.286 INFO  ProgressMeter -     SUPER_7:40280681             61.0             558840000        9158413.8

05:40:36.288 INFO  ProgressMeter -     SUPER_7:42426922             61.2             560476000        9160200.0

05:40:42.127 INFO  CombineGVCFs - Shutting down engine

[November 1, 2023 5:40:42 AM EDT] org.broadinstitute.hellbender.tools.walkers.CombineGVCFs done. Elapsed time: 61.32 minutes.

Runtime.totalMemory()=4901044224

java.lang.IllegalStateException: The elements of the input Iterators are not sorted according to the comparator htsjdk.variant.variantcontext.VariantContextComparator

at htsjdk.samtools.util.MergingIterator.next(MergingIterator.java:107)

at java.util.Iterator.forEachRemaining(Iterator.java:116)

at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)

at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)

at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)

at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)

at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)

at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)

at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)

at org.broadinstitute.hellbender.engine.MultiVariantWalker.traverse(MultiVariantWalker.java:136)

at org.broadinstitute.hellbender.engine.MultiVariantWalkerGroupedOnStart.traverse(MultiVariantWalkerGroupedOnStart.java:165)

at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1085)

at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:140)

at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:192)

at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:211)

at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)

at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)

at org.broadinstitute.hellbender.Main.main(Main.java:289)

CombineGVCFs gatk • 877 views

ADD COMMENT • link updated 12 months ago by GenoMax 147k • written 13 months ago by Matteo ▴ 10

score 3 · Accepted Answer · 2023-11-02

3

Entering edit mode

13 months ago

Pierre Lindenbaum 164k

the VCF must be sorted in the order of the chromosomes in the bPogPus1_combined_assembly.dict file (and in the ##contig= lines in the VCF header). A solution is to use https://gatk.broadinstitute.org/hc/en-us/articles/360036453432-SortVcf-Picard-

ADD COMMENT • link 13 months ago by Pierre Lindenbaum 164k

0

Entering edit mode

Thanks Pierre!! That worked!

ADD REPLY • link 12 months ago by Matteo ▴ 10

0

Entering edit mode

so validate my answer ( tick on the left )

ADD REPLY • link 12 months ago by Pierre Lindenbaum 164k