Entering edit mode
2.4 years ago
Marko
▴
20
Hello to all,
I am not from Bioinf field but there is no issue for me running bbduk trimming command :)
I was wondering is it possibile to load paired fastq reads directly form S3 bucket?
Even better, load input from S3 and stream the output to S3 directly as well = ]
I find many tools are "missing" this feature now that the cloud is "default" go to. Especially for storage.
I think this is because people prioritize solving problems that are objectively more difficult. Storage is not an issue for most people in this field, as one can buy a 4-10 Tb hard disk for well under $500.
Yes ofcorse. I did not mean to criticize. I am aware that my correspondence is blunt sometimes though :).
You can use a workflow orchestrator like nextflow - make sure to configure it with your aws stuff and then use nextflow's s3 storage support to write your workflow(s) - this way you can stream data right from S3 and write back to s3.
IMHO, using the fusion filesystem with Nextflow will be best as shown here
Hi, thanks for taking time to share. Since I don't come from bioinf side my question probably lacks context. Question is more of a "low level" nature. Lie techincal without using "third party" tools. Rather I asked it for educational purpose.
I will ask similar in a new thread hoping to make myself more clear.