Forum:Discussion: Uploading terabytes of data to NCBI SRA

Question

[James Reeve]

Large genomic datasets are becoming increasingly common, and often we need to find a place to archive and share data once our project is done. Probably the most widely used archive is NCBI's Short Read Archive (SRA). However, if you ever tired to use this for large data sets, their ftp upload option is a pain. There are frequent drop-outs and you only have a few minutes from logging in to navigate to your directory and start to upload. In short, I find data archiving a very frustrating experience.

I want to get a discussion going about the tricks and tips for streamlining this process. Especially, I'd like to know how one connects a remote server to NCBI.

Answer 1

[GenoMax]

If you have a need to upload (tens or more) terabytes of data to NCBI then you need to directly reach out to SRA support and work out a solution.

Otherwise using Aspera connect for upload should be the solution that is preferred: https://www.ncbi.nlm.nih.gov/sra/docs/submitfiles/ Uploads in theory would only be limited by bandwidth your institutions allows you to use with Aspera (since I would imagine that NCBI has access to larger networking pipes than you probably do).

Looks like they also provide uploads from Amazon S3 buckets.

Answer 2

[benformatics]

I've never had a problem with uploading to the SRA (both within and outside the US) for hundreds of GB of data - you might need to check with your internet provider if frequent drop-outs are a problem

[James Reeve]

I'm based at a field station, so drop-outs are unfortunately unavoidable. I want to use the remote server since it's based in a major city.

[GenoMax]

If your data is at located at the remote server then there should be no issues with dropouts. Sounds like the problem may be with your link from the field station (is the data being generated there) to the remote server?