Hi,
I am trying to setup PHG. I am using Singularity as I am working on a cluster.
The installation sis pretty simple as described on : https://bitbucket.org/bucklerlab/practicalhaplotypegraph/wiki/UserInstructions/CreatePHG_step0_singularity.md
So I ran:
$ cd phg_singularity
$ module load singularity
$ singularity pull docker://maizegenetics/phg
$ singularity pull docker://maizegenetics/phg_liquibase $singularity
$ build phg_22.simg docker://maizegenetics/phg:0.0.22
And followed up with building the default data structure:
$ cd ../phg_run
$ singularity exec -B /absolute/path/phg_singularity/:/phg/ /absolute/path/phg_singularity/phg_22.simg /tassel-5-standalone/run_pipeline.pl -debug -Xmx1G -MakeDefaultDirectoryPlugin -workingDir /phg/ -endPlugin
This ran without troubles. I followed with copying files into the proper directories and filled in the phg_run/inputDir/load_genome_data.txt and other key files.
The next step would be the setup of the initial PHG database: https://bitbucket.org/bucklerlab/practicalhaplotypegraph/wiki/UserInstructions/MakeInitialPHGDBPipeline.md
Sadly, all steps from here on are only documented for the Docker installation of PHG
I changed:
$ WORKING_DIR=local/directory/where/MakeDefaultDirectory/was/run/
$ DOCKER_CONFIG_FILE=/phg/config.txt
$ docker run --name create_initial_db --rm \
-v ${WORKING_DIR}/:/phg/ \
-t maizegenetics/phg:latest \
/tassel-5-standalone/run_pipeline.pl -Xmx100G -debug -configParameters ${DOCKER_CONFIG_FILE} \
-MakeInitialPHGDBPipelinePlugin -endPlugin
Into the Singularity version: (probably not quite right, but is seems to run somehow)
$ cd ../phg_run
$ DOCKER_CONFIG_FILE=config.txt
$ singularity exec \
-B /absolute/path/phg_singularity/:/phg/ /absolute/path/phg_singularity/phg_22.simg \
/tassel-5-standalone/run_pipeline.pl -Xmx100G -debug -configParameters ${DOCKER_CONFIG_FILE} \
-MakeInitialPHGDBPipelinePlugin -endPlugin
While this appears to run, it crashes with UNASSIGNED values of the config.txt file
An example of config.txt is given on: https://bitbucket.org/bucklerlab/practicalhaplotypegraph/wiki/UserInstructions/MakeInitialPHGDBPipeline.md
Yet the example file:
host=localHost
user=sqlite
password=sqlite
DB=/phg/phg_db_name.db
DBtype=sqlite
# Load genome intervals parameters
referenceFasta=/phg/inputDir/reference/Ref.fa
anchors=/phg/anchors.bed
genomeData=/phg/inputDir/reference/load_genome_data.txt
refServerPath=irods:/ibl/home/assemblies/
#liquibase results output directory, general output directory
outputDir=/phg/outputDir
liquibaseOutdir=/phg/outputDir
looks very different from the config file which was created in my file system:
########################################
#Required Parameters:
########################################
HaplotypeGraphBuilderPlugin.methods=**UNASSIGNED**
HaplotypeGraphBuilderPlugin.configFile=**UNASSIGNED**
CreateIntervalBedFilesPlugin.dbConfigFile=**UNASSIGNED**
CreateIntervalBedFilesPlugin.refRangeMethods=**UNASSIGNED**
GetDBConnectionPlugin.create=**UNASSIGNED**
GetDBConnectionPlugin.config=**UNASSIGNED**
LoadAllIntervalsToPHGdbPlugin.genomeData=**UNASSIGNED**
LoadAllIntervalsToPHGdbPlugin.outputDir=**UNASSIGNED**
LoadAllIntervalsToPHGdbPlugin.ref=**UNASSIGNED**
LoadAllIntervalsToPHGdbPlugin.anchors=**UNASSIGNED**
LoadHaplotypesFromGVCFPlugin.wgsKeyFile=**UNASSIGNED**
LoadHaplotypesFromGVCFPlugin.bedFile=**UNASSIGNED**
LoadHaplotypesFromGVCFPlugin.haplotypeMethodName=**UNASSIGNED**
LoadHaplotypesFromGVCFPlugin.gvcfDir=**UNASSIGNED**
LoadHaplotypesFromGVCFPlugin.referenceFasta=**UNASSIGNED**
FilterGVCFSingleFilePlugin.inputGVCFFile=**UNASSIGNED**
FilterGVCFSingleFilePlugin.outputGVCFFile=**UNASSIGNED**
FilterGVCFSingleFilePlugin.configFile=**UNASSIGNED**
RunHapConsensusPipelinePlugin.collapseMethod=**UNASSIGNED**
RunHapConsensusPipelinePlugin.dbConfigFile=**UNASSIGNED**
AssemblyHaplotypesMultiThreadPlugin.outputDir=**UNASSIGNED**
AssemblyHaplotypesMultiThreadPlugin.keyFile=**UNASSIGNED**
referenceFasta=**UNASSIGNED**
########################################
#Defaulted parameters:
########################################
HaplotypeGraphBuilderPlugin.includeSequences=true
HaplotypeGraphBuilderPlugin.includeVariantContexts=false
CreateIntervalBedFilesPlugin.windowSize=1000
CreateIntervalBedFilesPlugin.bedFile=intervals.bed
LoadHaplotypesFromGVCFPlugin.queueSize=30
LoadHaplotypesFromGVCFPlugin.mergeRefBlocks=false
LoadHaplotypesFromGVCFPlugin.numThreads=3
LoadHaplotypesFromGVCFPlugin.maxNumHapsStaged=10000
RunHapConsensusPipelinePlugin.minTaxa=1
RunHapConsensusPipelinePlugin.distanceCalculation=Euclidean
RunHapConsensusPipelinePlugin.minFreq=0.5
RunHapConsensusPipelinePlugin.minCoverage=0.1
RunHapConsensusPipelinePlugin.mxDiv=0.01
RunHapConsensusPipelinePlugin.clusteringMode=upgma
RunHapConsensusPipelinePlugin.maxClusters=30
RunHapConsensusPipelinePlugin.minSites=30
RunHapConsensusPipelinePlugin.maxThreads=1000
RunHapConsensusPipelinePlugin.kmerSize=7
AssemblyHaplotypesMultiThreadPlugin.mummer4Path=/mummer/bin/
AssemblyHaplotypesMultiThreadPlugin.loadDB=true
AssemblyHaplotypesMultiThreadPlugin.minInversionLen=7500
AssemblyHaplotypesMultiThreadPlugin.assemblyMethod=mummer4
AssemblyHaplotypesMultiThreadPlugin.entryPoint=all
AssemblyHaplotypesMultiThreadPlugin.numThreads=3
AssemblyHaplotypesMultiThreadPlugin.clusterSize=250
numThreads=10
Xmx=10G
picardPath=/picard.jar
gatkPath=/gatk/gatk
tasselLocation=/tassel-5-standalone/run_pipeline.pl
fastqFileDir=/tempFileDir/data/fastq/
tempFileDir=/tempFileDir/data/bam/temp/
dedupedBamDir=/tempFileDir/data/bam/DedupBAMs/
filteredBamDir=/tempFileDir/data/bam/filteredBAMs/
gvcfFileDir=/tempFileDir/data/gvcfs/
extendedWindowSize=1000
mapQ=48
#Sentieon Parameters. Uncomment and set to use sentieon:
#sentieon_license=**UNASSIGNED**
#sentieonPath=/sentieon/bin/sentieon
########################################
#Optional Parameters With No Default Values:
########################################
HaplotypeGraphBuilderPlugin.chromosomes=null
HaplotypeGraphBuilderPlugin.haplotypeIds=null
CreateIntervalBedFilesPlugin.extendedBedFile=null
LoadHaplotypesFromGVCFPlugin.haplotypeMethodDescription=null
RunHapConsensusPipelinePlugin.referenceFasta=null
RunHapConsensusPipelinePlugin.rankingFile=null
RunHapConsensusPipelinePlugin.collapseMethodDetails=null
AssemblyHaplotypesMultiThreadPlugin.gvcfOutputDir=null
#FilterGVCF Parameters. Adding any of these will add more filters.#exclusionString=**UNASSIGNED**
#DP_poisson_min=0.0
#DP_poisson_max=1.0
#DP_min=**UNASSIGNED**
#DP_max=**UNASSIGNED**
#GQ_min=**UNASSIGNED**
#GQ_max=**UNASSIGNED**
#QUAL_min=**UNASSIGNED**
#QUAL_max=**UNASSIGNED**
#filterHets=**UNASSIGNED**
This is where I cannot figure out, what to fill in into the UNASSIGNED sections. I assume the file looks different because of the Singularity setup I ran ? Can you give me an example for this file ?
It would be also very helpful if you could add a singularity version of most commands shown in the Wiki, as they do seem very different from the Docker commands...
Cheers Jakob
Hello,
The main reason that it looks a lot different is that MakeDefaultDirectoryPipelinePlugin attempts to create a config file based on all of the parameters available to all the plugins that are run for the pipelines. It has a lot of extra parameters. The parameters marked with
**UNASSIGNED**are not automatically filled as they are mostly parameters you should set or file names of your specific files. That being said, the documentation should be a lot better with respect to the configuration parameters and have more clear examples.If you replace all of the
**UNASSIGNED**parameters in your config with the following it should work better.If you are loading in Assemblies only as haplotypes, you should remove any of the lines starting with LoadHaplotypesFromGVCFPlugin. And if you are creating haplotypes based on WGS only, remove the lines starting with AssemblyHaplotypesMultithreadPlugin. You will need to set referenceFasta and LoadAllIntervalsToPHGdbPlugin.ref to match the names of your files otherwise it will throw more errors.
I will update the documentation here to include a simple suggested config file. We will also look into adding in singularity instructions as well.
Thanks, Zack Miller
Hi Zack,
Thanks a lot.
I started to fill in the gaps, but there are still plenty of spaces / UNASSIGED sections to fill.
I figured that for the initial run I mainly need to setup the required parameters: (this is how my config.txt looks now):
Again it is still very much not even close to the same file you hav, but some lines do match. This is probably because singularity .
I am mainly struggleing to find these plugin configs.
I assume the plugins are sitting somewhere int he singularity containers ? Or are these plugins somewhere within the working directory ?
Cheers Jakob
Hello,
I mentioned to replace all of the UNASSIGNED ones with what I had posted. I removed a lot of the specific PluginParameters where they had shared parameters with other plugins.
TASSEL(which the PHG is based off of) allows you to just specify the shared parameter name. An example of this is referenceFasta being shared by a number of plugins. You could specify Plugin1.referenceFasta, Plugin2.referenceFasta and so on, but for your use they are all the same. Some of the parameters have good defaults as well so they could be removed.
The config file has nothing to do with singularity.
The plugins are sitting in the Docker container(and by extension the singularity container) as part of TASSELs lib folder(/tassel-5-standalone/lib/phg.jar). When you run
/tassel-5-standalone/run_pipeline.pl -MakeDefaultDirectoryPlugin...you are actually running a plugin which is in the container.Hi Zack,
Yes singularity has nothing to do with this. It looks like it all stands and falls with a proper config file.
I hacked the config file a lot and got some steps further into the pipeline. I replaced the #required parameters section with the part you wrote and I added the sql db config section to the file as well, leading to this pretty hacky version:
Then I created an arbitrary interval range file with 100 x 1000bp ranges. This let me run the MakeInitialPHGDBPipeline Plugin, partially.
these completed succesfully: GetDBConnectionPlugin LoadAllIntervalsToPHGdbPlugin
This one failed: LiquibaseUpdatePlugin
I initialy ran it with the docker image phg:latest. This ran into an error: Database too old (v 0.0.10) please update, which then fails because it's too old to update via Liquibase.
Then I pulled a newer docker image from the nightly build phg:0.026. It created a newer database (v 0.0.24), but then was incompatible with phg v0.026 (I assume thats a minor bug?)
So I pulled version 0.024 from docker which completed the MakeInitialPHGDBPipeline Plugin (still couldn't run Liquibase, but finished).
But the database seems still corrupt. I made GVCF's, a haplotype keyfile and a CreateHaplotypesFromFastq.groovy config file (which is the combination of the host options and the following haplotype options):
then I run the CreateHaplotypesFromFastq.groovy pipelime using singularity: singularity exec -B ./:/phg/ /group/pawsey0149/jpetereit/PHG/phg_24.simg /CreateHaplotypesFromGVCF.groovy -config haplotype_config.txt
It runs for a bit, but fails with the following error:
[DefaultDispatcher-worker-2] INFO net.maizegenetics.pangenome.db_loading.LoadHaplotypesFromGVCFPlugin - Done setting up variables for file /phg/inputDir/gvcf/SRR9969480.g.vcf. Moving on to processing reference ranges. [pool-1-thread-1] DEBUG net.maizegenetics.plugindef.AbstractPlugin - Error writing to the DB:
Which I assume is because of too many hacks from my side.
I also get pretty confused with the config file and the parameters, there are just soo many.
Can you doublecheck what happens with these DB versions and the updates ?
Would you normally run the whole pipeline with a singular config file ? Is it maybe worth making a specific config file for each pipeline step ?
Can you upload an example of a complete config file ?
Hi,
I'll add the CreateHaplotypesFromGVCF log in the next days. It did work in the end. And I probably had some mismatching bed files.
The out of memory problem seems very odd. I have a 430mb genome (rice) and 4 Taxons. And I only added 1000 x 1000bp ranges. Nothing spectacular.
I found the problem.
CreateConsensi.sh looks for Xmx in the config file which I did not set, so it defaulted back to 1350mb.
I added
to the config, which now seems not to break anymore.
I'll add more to this thread once everything works :)
Cheers J
Hi everyone,
Newbie here. I'm trying to run the CreateHaplotypesFromGVCF plugin here and I keep getting the following error:
I saw user: petinho86 mention about the error: couldn't write to the db. Was the error the same as I have? If yes, was the only problem with bedfile and nothing else? I tried using the same bed file that I used for the db but that doesn't seem to help. I get the same error. Please let me know
Thanks B
Please don't add questions as answers to a pre-existing thread. If your problem is different then please start a new thread.