How to scrape BioMart data from https://sorfs.ugent.be/ website

Question

[QX]

Hi all,

I would like to download database from this website: https://sorfs.ugent.be/database but it seems that they do not maintain the database anymore, only the table available.

Is there anyway to pull the table from this website as a csv file?

Best,

[Pierre Lindenbaum]

did you ask the authors to restore their server ?

[QX]

yes but they did not reply

Answer 1

[Pierre Lindenbaum]

something like:

seq 1 100 4377380 | while read F ; do wget   --no-check-certificate  -O - "https://sorfs.ugent.be/database/micropep?sEcho=4&iColumns=18&sColumns=%2C%2C%2C%2C%2C%2C%2C%2C%2C%2C%2C%2C%2C%2C%2C%2C%2C&iDisplayStart=${F}&iDisplayLength=100&mDataProp_0=0&sSearch_0=&bRegex_0=false&bSearchable_0=true&bSortable_0=true&mDataProp_1=1&sSearch_1=&bRegex_1=false&bSearchable_1=true&bSortable_1=true&mDataProp_2=2&sSearch_2=&bRegex_2=false&bSearchable_2=true&bSortable_2=true&mDataProp_3=3&sSearch_3=&bRegex_3=false&bSearchable_3=true&bSortable_3=true&mDataProp_4=4&sSearch_4=&bRegex_4=false&bSearchable_4=true&bSortable_4=true&mDataProp_5=5&sSearch_5=&bRegex_5=false&bSearchable_5=true&bSortable_5=true&mDataProp_6=6&sSearch_6=&bRegex_6=false&bSearchable_6=true&bSortable_6=true&mDataProp_7=7&sSearch_7=&bRegex_7=false&bSearchable_7=true&bSortable_7=true&mDataProp_8=8&sSearch_8=&bRegex_8=false&bSearchable_8=true&bSortable_8=true&mDataProp_9=9&sSearch_9=&bRegex_9=false&bSearchable_9=true&bSortable_9=true&mDataProp_10=10&sSearch_10=&bRegex_10=false&bSearchable_10=true&bSortable_10=true&mDataProp_11=11&sSearch_11=&bRegex_11=false&bSearchable_11=false&bSortable_11=true&mDataProp_12=12&sSearch_12=&bRegex_12=false&bSearchable_12=false&bSortable_12=true&mDataProp_13=13&sSearch_13=&bRegex_13=false&bSearchable_13=false&bSortable_13=true&mDataProp_14=14&sSearch_14=&bRegex_14=false&bSearchable_14=false&bSortable_14=true&mDataProp_15=15&sSearch_15=&bRegex_15=false&bSearchable_15=false&bSortable_15=true&mDataProp_16=16&sSearch_16=&bRegex_16=false&bSearchable_16=false&bSortable_16=true&mDataProp_17=17&sSearch_17=&bRegex_17=false&bSearchable_17=false&bSortable_17=true&sSearch=&bRegex=true&iSortCol_0=0&sSortDir_0=asc&iSortingCols=1&_=1716563877578" | python -m json.tool ; done

{
    "aaData": [
        [
            "<a href = 'database/andreev_2015:100170' >andreev_2015:100170</a>",
            "<a href='http://www.ensembl.org/Multi/Search/Results?q=ENST00000405356;y=0;site=ensembl_all;x=0;page=1;facet_feature_type=Gene'>ENST00000405356</a>"
,
            "human",
            "andreev_2015",
            "10",
            102157437,
            102157610,
            58,
            "exonic",
            "protein_coding",
            "yes",
            "0.4749",
            "1.5047",
            "MPSESVCLLGRLQPKHQRVAAVKSPVMMMMRRTKRNSLSRFAALGRKRGLRIRKEET*",
            "CTGCCAAGCGAGTCGGTCTGCCTCCTGGGAAGGCTGCAGCCAAAGCATCAGAGAGTAGCAGCAGTGAAGAGTCCAGTGATGATGATGATGAGGAGGACCAAAAGAAACAGCCTGTCCAGGTTTGCAGCTTTGGGAAGAAAAAGGGGTT
TAAGGATTAGAAAGGAAGAAACCTAA",
            "0.357142857143",
            "-9.23678239084",
            "ENSG00000166197"
        ],
        [
            "<a href = 'database/andreev_2015:100182' >andreev_2015:100182</a>",
            "<a href='http://www.ensembl.org/Multi/Search/Results?q=ENST00000405356;y=0;site=ensembl_all;x=0;page=1;facet_feature_type=Gene'>ENST00000405356</a>"
,
            "human",
            "andreev_2015",
            "10",
            102158107,
            102158169,
            21,
            "exonic",
            "protein_coding",
            "yes",
            "0.8899",
            "2.4239",
            "MILILTQAPRMSHQRTRSQR*",
            "CTGATTCTGATTCTGACTCAAGCTCCGAGGATGAGCCACCAAAGAACCAGAAGCCAAAGATAA",
            "0.368421052632",
            "-7.55004712725",

[QX]

thank you so much!

[Pierre Lindenbaum]

Don't forget to follow up on your threads. If an answer was helpful, you should upvote it; if the answer resolved your question, you should mark it as accepted. You can accept more than one answer if they all work. If an answer was not really helpful or did not work, provide detailed feedback so others know not to use that answer.