Run nextflow sections sequentially

Question

[Raygozak]

Hi, I'm running a nextflow pipeline and I have a limit of jobs i can submit to the cluster, however nextflow submits jobs for different sections of a pipeline as soon as the required files are available. Is there a way to tell it to fits submit all the tasks for a section (e.g. aligning) without trying to run the next step right away?

Thanks

Please read the question and answer what is asked. I don't need any notes on what i'm doing.

[ATpoint]

You can make each alignment process emit a value, then gather/concat this value from all alignment jobs into a new value/variable and let the downstream process read that new one as input. It's not actually used by the downstream process for any operation other than just serving as a serialization instance. The new processes won't start until all alignmrnt jobs are finished. Not aware of an in-built method.

[Raygozak]

this doesn't answer the question asked, nor is related in the least.

[ATpoint]

It answers the question exactly, and I would appreciate a different tone. Consider paying a consultant in case a free online community is not serving you appropriately.

[Pierre Lindenbaum]

this doesn't answer the question asked, nor is related in the least.

hum... yes it is. My answer below is an illustration of what ATpoint said.

[Ram]

Why did you delete this post, Raygozak?

Answer 1

[Pierre Lindenbaum]

workflow {
    mulitple_ch = ....
    step_ch = MANY(mulitple_ch)
    WAIT_FOR_MANY(step_ch.output.collect())
}

[Raygozak]

Does the word running has a different meaning of what i'm thinking?

[colindaven]

@Raygozak Please be nice to the people who have provided you with answers and have some humility. I understand exactly what they mean, and have learned something here and or already use this pattern in my pipelines. If you do not, then it is not their fault, but yours.