Making A Bioinformatics Application Available In The Amazon Ec2 Cloud

Question

[Jlw]

I have a user who is interested in having their application (custom c++/python application) hosted in the Amazon cloud and made accessible to the public. Configuration of the environment is sometimes difficult for users so this would make a VM immediately available to users. Use of the AWS Marketplace (via creation of an AMI) seems to be one method to do this. There is an option to host for free and allow unlimited access to the public for free (aside from the individuals own access to the cloud). There are some security regulations that lists limiting users ability to input their own data to the application (which in this case would be required). It also states this is limited to single CPU operations. Does anyone have experience hosting a non-web based application. What are the best approaches for academic software?

We are also open to looking outside of Amazon for the task if it would be better to use Google compute...or?

[Pierre Lindenbaum]

what about asking an academic lab to host your application ?

[Jlw]

Ah, it is already hosted on a website with download/installation instructions through an academic group. We are hoping to use this as another option for access

[Malachi Griffith]

This tutorial post might be useful: Post not found

Answer 1

[Matt Shirley]

Make it a web application. As long as the data are not too large, you can have the user upload a file, then have the server process run your application and return the result as a web page. If you don't have access to a web server in your lab, I would recommend DigitalOcean for the price and performance. Take a look at this example: http://tripod.mattshirley.com. It's something I created this year, and is by no means the best possible web application. Briefly it:

1. Runs a Flask framework WSGI service that renders the site templates, and provides REST APIs
2. When a user uploads data, the data are validated for consistency, checked for illegal characters
3. If the data look safe and correct, the service sends a request to run the triPOD script to a Celery queue, which is limited to running X jobs at once
4. The user is sent to a page that polls an API that returns success when the job is finished and results are ready
5. Results are then retrieved using the Celery ID and rendered in a results template page

You could do something similar for your program, and since it's written partly in Python, Flask might also be a good framework for you to work with. An (outdated) version of the app I created is on Github: https://github.com/mdshw5/tripod-web.

[Damian Kao]

+1 for making a web app. However, it's not that easy to put it all together. I prefer ngninx + bottle.py stack. I've never used Celery. It seems really useful.

[Jlw]

Thanks! That does look like a good option though there is no budget to pay for hosting (even though this cost is minimal) so it would have to be in an environment where users pay for time/space on the cloud but the application is available for free (my understanding of how marketplace can work).

Answer 2

[Giovanni M Dall'Olio]

You can make the app available as a docker container.

Advantages:

• users will be able to run it on their computer or anywhere else, not only on amazon
• It is still a virtual image, so users don't have to worry about configuration and installation of software
• It should be easier to prepare than an amazon VM. You only have to write a text file.

Some links:

• docker
• biodocker
• oswitch (an application of docker from which you can take inspiration)
• an article on my blog explaining what I understood of docker
• Performances of docker on a HPC cluster

Answer 3

[Istvan Albert]

Your wording seems to imply that you want to allow access to the public to log in and use the program as a standalone application.

To which I say categorically no, you should not consider that option, your system will be hijacked and used for all kinds of other purposes.

[Jlw]

The idea is to host this on the Amazon cloud (not our own system) similar to how you would access other bioinformatics applications, such as an assembler or SNP caller. We are just trying to find the best medium to serve out this application.

[Istvan Albert]

it does not matter where you host it, the idea of having a computer with free public access where users can log into a shell to run programs is not recommended by any means

Answer 4

[John]

Programs which were developed to be run by a trusted user cannot be easily ported over to the web, unfortunately. Anything that reads/writes to disk is going to be a security concern. Double or triple the risk ratio if the server-side code is also publicly available as part of an academic publication.

Of course, it can be done - but if you are not provided the budget even for hosting, i doubt you will be allowed to commit the resources required to maintain and secure a public-facing server.

Whats the worst that could happen? Well, in descending probability:
- The server will be used as a jumping off point for other more 'noisy' attacks, safe in the knowledge that if 100 totally unconnected people visit your site a day, tracking the actual attacker will be next to impossible.
- The website will be defaced by "p|-|ysici5ts_rul3" to contains images and text of how superior physics is as a natural science. or something...
- The website will start pushing "required java binaries" to users of the site.
- Ads appear everywhere.

I learnt every single one of these problems the hard way - and I am just supremely grateful that in all instances I noticed something was going on before someone from GCHQ came knocking on my door.