Hi All. I'd appreciate any maintainable solutions for storing genomic data when it scales to tens to hundreds of terabytes of data. E.g. external harddrives, cloud storage. Any comments on what you guys are using and why would be great. Thanks in advance!
I second tape. You can fit 4-6TB per LTO-6 tape with hardware compression.
But if you don't have tape infrastructure available and are starting from scratch then the cheapest archival cloud storage (google coldline. amazon glacier) could work. This would assume you would basically not be accessing that data regularly to make it cost effective.
Theoretically, you could store the data in CyVerse, which is even cheaper than tape for academic use:
CyVerse's cloud-based data storage is optimized for large data, free to most scientific researchers, accessible through multiple interfaces, and leaves access control in the hands of the data owners.
From: http://www.cyverse.org/data-policy
As with all cloud services, the service could shut down or change pricing or just lose/corrupt your files at any point.
Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
version 2.3.6
Tape. Because it's cheap.
Compression and encoding. Algorithms will always beat hardware.