How To Write Bash Script To Download Files From Specific Databse

Question

[jack]

Hi all,

I want to download some files from online database. but it does not allow to download all the files at once, instead it's offer to download file for searched keyword, and because my keywords are more than 20000, so in principal it's not feasible for me for example, I want to download whole information about miRNA-mRNA interaction from SarBase but it does not offer option to download all of them at once. I wonder , how can I download it by writing some scripts. can anybody help me ?

[Pierre Lindenbaum]

how about http://starbase.sysu.edu.cn/downloads/ ?

Answer 1

[Alex Reynolds]

For static files, you could use either wget or curl in a bash script, e.g.:

#!/bin/bash
wget https://www.example.com/foo/bar.baz -O /path/to/destination/bar.baz
curl https://www.example.com/baz/foo.bar --output /path/to/destination/foo.bar
...

If it is a web form, you could emulate submitting a form with curl -F and key-value pairs representing form fields, using the URL or address of the form CGI, e.g.:

#!/bin/bash
curl -F "foo=bar;baz=foo" https://www.example.com/some_form.cgi --output /path/to/destination/some_result

You can also use wget --post-data; take a look at the man pages for the two tools.

If you put textual form data into a script, make sure its contents are percent-encoded. If you put binary data into a script, you'll need to do some other encoding, such as base64-encoding. There are numerous CLI tools to do these and other kinds of encoding.

Answer 2

[Zhaorong]

Is this what you want? http://starbase.sysu.edu.cn/browseClipSeq.php Click "export" at the upper left corner of the table.

Answer 3

[zhxq09]

In miRNA-mRNA query webpage, if you do not input/select anything, you can browse and download all miRNA-mRNA interactions. You only click "export" at the upper left corner of the table and download all results.