Nextflow - How to pass the yml format input file from an argument to channel as a list

Question

[kani]

How to pass the yml format input file from an argument to a channel samples as a list? Appreciate if any nextflow function or simple solution to make the channel samples works with --input_list s3://my-bucket/params.yml instead of -params-file s3://my-bucket/params.yml

The given below channel samples code works with -params-file s3://my-bucket/params.yml but that's not I'm looking.

$ nextflow run main.nf \ -config nextflow.config \ --input_list s3://my-bucket/params.yml \ --publish_dir ./output

--input_list s3://my-bucket/params.yml

samples:
-
 biosample_id: WGS001
 bam: WGS001.bam
-
 biosample_id: WGS0002
 bam: NWGS0002.bam

workflow

Channel
    .fromList( params.samples )
    .ifEmpty { ['biosample_id': params.biosample_id, 'bam': params.bam] }
    .set { samples }


Channel
    samples.branch { rec ->
    ....

Channel
    samples.map { it.biosample_id }
    ....

Answer 1

[Pierre Lindenbaum]

I don't think there is a native NF operator to read YAML:

Nevertheless, nextflow itself contains a java library parsing yaml. org.yaml.snakeyaml.Yaml

So you can try:

	params.samples="NO_FILE"
	workflow {
	ch = Channel.fromPath(params.samples).
	flatMap(F->{
	def yaml = new org.yaml.snakeyaml.Yaml();
	def map = yaml.load(F);
	return map.samples;
	});
	doIt(ch)
	}
	process doIt {
	tag "${row.biosample_id} = ${row.bam}"
	input:
	val(row)
	script:
	"""
	echo "${row.biosample_id} = ${row.bam}"
	"""
	}

view raw main.nf hosted with ❤ by GitHub

	samples:
	-
	biosample_id: WGS001
	bam: WGS001.bam
	-
	biosample_id: WGS0002
	bam: NWGS0002.bam

view raw test.yaml hosted with ❤ by GitHub

Answer 2

[pbioinf]

Use the undocumented -params-file option. Documentation is unfortunately lacking. The parsed yaml file becomes part of your parameters list in params.

From nextflow run -help:

-params-file
   Load script parameters from a JSON/YAML file

[Marcel Ribeiro-Dantas]

How come undocumented? https://www.nextflow.io/docs/latest/cli.html?highlight=params%20file

There are even examples over there.

[pbioinf]

You are right, last time I looked into this it was still undocumented.

[kani]

I want to pass the input file through an argument unfortunately not with params-file, reason being unable to use the params-file in the nftower dashboard under Pipeline parameters field which is equivalent to params-file option allowing to paste the json/yaml file in the field but it has a charcters limitation to give large number of samples list. To overcome this I need to give the input file with an arg like. insput_list: s3://my-bucket/params.yaml

[Maxime Garcia]

I'd recommend to use a csv file to give out input file instead. This is what we do most of the time in nf-core nowadays.

Answer 3

[kani]

This works well...

import groovy.yaml.YamlSlurper

params.inputs_list = "inputs.yaml"

workflow {    
    inputs = new YamlSlurper().parse(file(params.inputs_list))

    Channel
        .fromList(inputs['samples'])
        .ifEmpty { ['biosample_id': params.biosample_id, 'aln': params.aln] }
        .set { samples }

https://github.com/nextflow-io/nextflow/discussions/4288