Using Amazon Web Services?

Question

[Peter]

Hello,

I am trying to use amazon web services for bioinformatics. I created a linux EC2 instance, but I've found that the space is limited to 10gB. I am attempting to take advantage of the EBS to create a larger hard drive space for me to work with. I am using the AWS console to attach the EBS to EC2 instance, but cannot figure out how to use the attached storage. Can anyone help me with the steps to make this all work?

Thanks!

[Istvan Albert]

10GB space sounds woefully inadequate to any modern computer. Are you sure you have it right? Check here.

[Malachi Griffith]

This tutorial post might be useful: Post not found

Answer 1

[Alex]

Useful EC2 tutorials for bioinformatics:

1. Getting started guide: http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/
2. Getting started JAVA: http://aws.amazon.com/articles/Amazon-EC2/3586
3. Getting started Python: http://aws.amazon.com/articles/Amazon-EC2/3998
4. EBS (this is MySQL example, but it is very useful and solves your current problems): http://aws.amazon.com/articles/Amazon-EC2/1663
5. Hadoop tutorial: http://aws.amazon.com/articles/Amazon-EC2/873

To check available space try this:

[root@domU-12-31-39-00-B2-01 ~]# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             10321208   1636668   8160252  17% /
/dev/sda2            153899044    192072 145889348   1% /mnt

10Gb is the size of root disk, "other" disk space is in the /mnt directory.

For data analysis I usually use instance disk space ~1Tb. Note that all data will be deleted when you delete instance. And I save all important data to EBS (permanent storage). If you use US zone then network connection with NCBI is fantastic so expensive data keeping in permanent EBS is not necessary. Alternatively you can use public bioinformatics EBS, see full list here http://aws.amazon.com/datasets?_encoding=UTF8&jiveRedirect=1

Available biological public EBS:

1. 1000 genomes
2. Human Liver Cohort (Sage Bionetworks)
3. C57BL/6J by C3H/HeJ Mouse Cross (Sage Bionetworks)
4. Ensembl Annotated Human Genome Data - for FASTA
5. Ensembl Annotated Human Genome Data - for MySQL
6. Illumina - Jay Flatley (CEO of Illumina) Human Genome Data Set
7. GenBank
8. YRI Trio Dataset
9. Ensembl - FASTA Database Files
10. Unigene
11. and more

[Michael Schubert]

Just one suggestion: df -h would be more readable :)

[Mndoci]

Additionally, if you boot of an EBS-backed image, you can have up boot volumes up to 10 TB

[Mndoci]

Just realized I wrote 10 TB .. make that 1 TB

Answer 2

[Marina Manrique]

Maybe you just have to mount the EBS volume,

Here you can get very basics instructions to mount an attached volume to the instance.

Once you've attached the volume to the instance you just have to mount it with "mount" command

HTH

Marina

Answer 3

[Casbon]

If you use an instance store based volume, rather than a EBS backed one, you will have ~300GB to play with at /mnt. However, it will not be persisted when the instance is deallocated.

Since you have 10GB, you are on EBS. This means you will need to attach a larger EBS at /dev/sdx and mount it as Marina points out.