Nextflow - How to pass the yml format input file from an argument to channel as a list

Question

[kani]

How to pass the yml format input file from an argument to a channel samples as a list? Appreciate if any nextflow function or simple solution to make the channel samples works with --input_list s3://my-bucket/params.yml instead of -params-file s3://my-bucket/params.yml

The given below channel samples code works with -params-file s3://my-bucket/params.yml but that's not I'm looking.

$ nextflow run main.nf \ -config nextflow.config \ --input_list s3://my-bucket/params.yml \ --publish_dir ./output

--input_list s3://my-bucket/params.yml

samples:
-
 biosample_id: WGS001
 bam: WGS001.bam
-
 biosample_id: WGS0002
 bam: NWGS0002.bam

workflow

Channel
    .fromList( params.samples )
    .ifEmpty { ['biosample_id': params.biosample_id, 'bam': params.bam] }
    .set { samples }


Channel
    samples.branch { rec ->
    ....

Channel
    samples.map { it.biosample_id }
    ....

Answer 1

[Pierre Lindenbaum]

I don't think there is a native NF operator to read YAML:

Nevertheless, nextflow itself contains a java library parsing yaml. org.yaml.snakeyaml.Yaml

So you can try:

Answer 2

[pbioinf]

Use the undocumented -params-file option. Documentation is unfortunately lacking. The parsed yaml file becomes part of your parameters list in params.

From nextflow run -help:

-params-file
   Load script parameters from a JSON/YAML file

[Marcel Ribeiro-Dantas]

How come undocumented? https://www.nextflow.io/docs/latest/cli.html?highlight=params%20file

There are even examples over there.

[pbioinf]

You are right, last time I looked into this it was still undocumented.

[kani]

I want to pass the input file through an argument unfortunately not with params-file, reason being unable to use the params-file in the nftower dashboard under Pipeline parameters field which is equivalent to params-file option allowing to paste the json/yaml file in the field but it has a charcters limitation to give large number of samples list. To overcome this I need to give the input file with an arg like. insput_list: s3://my-bucket/params.yaml

[Maxime Garcia]

I'd recommend to use a csv file to give out input file instead. This is what we do most of the time in nf-core nowadays.

Answer 3

[kani]

This works well...

import groovy.yaml.YamlSlurper

params.inputs_list = "inputs.yaml"

workflow {    
    inputs = new YamlSlurper().parse(file(params.inputs_list))

    Channel
        .fromList(inputs['samples'])
        .ifEmpty { ['biosample_id': params.biosample_id, 'aln': params.aln] }
        .set { samples }

https://github.com/nextflow-io/nextflow/discussions/4288