can picard use multiple threads?

Question

[naruto1nrt]

Is there an argument can make Picard multiple threads?

In the picard home page, I didn't find it.

[Martombo]

I think it's implemented only for very few functions (like IlluminaBasecallsToFastq). which tool do you want to use?

Answer 1

[Pierre Lindenbaum]

yes, but not all:

	$ for F in /commun/data/packages/picard-tools-1.104/[A-Z]*.jar; do echo ":: $F" && java -jar ${F} 2>&1 | grep -i -E '(thread|parall|processor)'; done
	:: /commun/data/packages/picard-tools-1.104/AddOrReplaceReadGroups.jar
	:: /commun/data/packages/picard-tools-1.104/BamIndexStats.jar
	:: /commun/data/packages/picard-tools-1.104/BamToBfq.jar
	:: /commun/data/packages/picard-tools-1.104/BuildBamIndex.jar
	:: /commun/data/packages/picard-tools-1.104/CalculateHsMetrics.jar
	:: /commun/data/packages/picard-tools-1.104/CheckIlluminaDirectory.jar
	:: /commun/data/packages/picard-tools-1.104/CleanSam.jar
	:: /commun/data/packages/picard-tools-1.104/CollectAlignmentSummaryMetrics.jar
	:: /commun/data/packages/picard-tools-1.104/CollectGcBiasMetrics.jar
	:: /commun/data/packages/picard-tools-1.104/CollectInsertSizeMetrics.jar
	:: /commun/data/packages/picard-tools-1.104/CollectMultipleMetrics.jar
	:: /commun/data/packages/picard-tools-1.104/CollectRnaSeqMetrics.jar
	:: /commun/data/packages/picard-tools-1.104/CollectTargetedPcrMetrics.jar
	:: /commun/data/packages/picard-tools-1.104/commons-jexl-2.1.1.jar
	:: /commun/data/packages/picard-tools-1.104/commons-logging-1.1.1.jar
	:: /commun/data/packages/picard-tools-1.104/CompareSAMs.jar
	:: /commun/data/packages/picard-tools-1.104/CreateSequenceDictionary.jar
	:: /commun/data/packages/picard-tools-1.104/DownsampleSam.jar
	:: /commun/data/packages/picard-tools-1.104/EstimateLibraryComplexity.jar
	:: /commun/data/packages/picard-tools-1.104/ExtractIlluminaBarcodes.jar
	NUM_PROCESSORS=Integer Run this many PerTileBarcodeExtractors in parallel. If NUM_PROCESSORS = 0, number of
	NUM_PROCESSORS < 0 then the number of cores used will be the number available on the
	machine less NUM_PROCESSORS. Default value: 1. This option can be set to 'null' to clear
	:: /commun/data/packages/picard-tools-1.104/ExtractSequences.jar
	:: /commun/data/packages/picard-tools-1.104/FastqToSam.jar
	:: /commun/data/packages/picard-tools-1.104/FilterSamReads.jar
	:: /commun/data/packages/picard-tools-1.104/FixMateInformation.jar
	Exception in thread "main" net.sf.picard.PicardException: Must specify either an explicit OUTPUT file or a single INPUT file to be overridden.
	:: /commun/data/packages/picard-tools-1.104/IlluminaBasecallsToFastq.jar
	NUM_PROCESSORS=Integer The number of threads to run in parallel. If NUM_PROCESSORS = 0, number of cores is
	automatically set to the number of cores available on the machine. If NUM_PROCESSORS < 0,
	NUM_PROCESSORS. Default value: 0. This option can be set to 'null' to clear the default
	:: /commun/data/packages/picard-tools-1.104/IlluminaBasecallsToSam.jar
	NUM_PROCESSORS=Integer The number of threads to run in parallel. If NUM_PROCESSORS = 0, number of cores is
	automatically set to the number of cores available on the machine. If NUM_PROCESSORS < 0,
	NUM_PROCESSORS. Default value: 0. This option can be set to 'null' to clear the default
	:: /commun/data/packages/picard-tools-1.104/IntervalListTools.jar
	Exception in thread "main" java.lang.NullPointerException
	:: /commun/data/packages/picard-tools-1.104/MakeSitesOnlyVcf.jar
	:: /commun/data/packages/picard-tools-1.104/MarkDuplicates.jar
	:: /commun/data/packages/picard-tools-1.104/MarkIlluminaAdapters.jar
	:: /commun/data/packages/picard-tools-1.104/MeanQualityByCycle.jar
	:: /commun/data/packages/picard-tools-1.104/MergeBamAlignment.jar
	:: /commun/data/packages/picard-tools-1.104/MergeSamFiles.jar
	USE_THREADING=Boolean Option to create a background thread to encode, compress and write to disk the output
	file. The threaded version uses about 20% more CPU and decreases runtime by ~20% when
	:: /commun/data/packages/picard-tools-1.104/MergeVcfs.jar
	:: /commun/data/packages/picard-tools-1.104/NormalizeFasta.jar
	:: /commun/data/packages/picard-tools-1.104/picard-1.104.jar
	:: /commun/data/packages/picard-tools-1.104/QualityScoreDistribution.jar
	:: /commun/data/packages/picard-tools-1.104/ReorderSam.jar
	:: /commun/data/packages/picard-tools-1.104/ReplaceSamHeader.jar
	:: /commun/data/packages/picard-tools-1.104/RevertSam.jar
	:: /commun/data/packages/picard-tools-1.104/sam-1.104.jar
	:: /commun/data/packages/picard-tools-1.104/SamFormatConverter.jar
	:: /commun/data/packages/picard-tools-1.104/SamToFastq.jar
	:: /commun/data/packages/picard-tools-1.104/snappy-java-1.0.3-rc3.jar
	:: /commun/data/packages/picard-tools-1.104/SortSam.jar
	:: /commun/data/packages/picard-tools-1.104/SplitVcfs.jar
	:: /commun/data/packages/picard-tools-1.104/tribble-1.104.jar
	:: /commun/data/packages/picard-tools-1.104/ValidateSamFile.jar
	:: /commun/data/packages/picard-tools-1.104/variant-1.104.jar
	:: /commun/data/packages/picard-tools-1.104/VcfFormatConverter.jar
	:: /commun/data/packages/picard-tools-1.104/ViewSam.jar

view raw biostars-105499.sh hosted with ❤ by GitHub

Answer 2

[l0o0]

I have the same problem. And finally found solutions in picard faq.

You can not set number of threads for some picard packages, but you can set threads number for java

Try to the code below:

java -XX:ParallelGCThreads=<num of  threads> -jar <picard-package-name>.jar

I hope this will help.

[Zhilong Jia]

It seems the Elapsed times have no difference.

[haizi.zh]

I don't think this option turns on multithreading for picard. In fact, -XX:ParallelGCThreads is just about how JVM implements the garbage collection. It may have some impact on the overall performance, but the argument itself by no means has anything to do with picard multithreading.

Here's an in-depth introduction to parallel GC in JVM: https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/parallel.html

Answer 3

[Benedek Dankó]

You can use for example GNU parallel. Something like this:

ls *.bam | sed 's/.bam.*//' | parallel --eta -j <num of threads> "java -jar picard <package> I={}.bam O={}_out.bam"

[szp770]

bash shows "parallel: command not found"

[Pierre Lindenbaum]

https://www.google.com/search?client=firefox-b-d&q=command+not+found