Question

Error while downloading data from database[EGA]

2

Entering edit mode

6.7 years ago

Gene_MMP8 ▴ 240

I have access to a particular dataset in EGA. I am trying to download it. But I always get cut off from the server while downloading. The file size is 121.7 GB. Below is the error I am getting. (The error occurs after downloading 35GB)

Starting transfer loop for 08760ef0-82f9-4431-ac96-37f72843acf2 (true)

Download Error occurred at Stage 4
Error: Read timed out
Downloaded :
Error requesting this!

Can someone please point out the problem?

next-gen sequencing • 5.9k views

ADD COMMENT • link updated 6.2 years ago by miaowzai ▴ 390 • written 6.7 years ago by Gene_MMP8 ▴ 240

0

Entering edit mode

We are having similar issue with downloads failing after a prolonged wait time in downloading data from EGA servers. Using EGA Download Client and following command to request and download data#### Command

#Request
java -jar /apps/ega/download/default.jar -pf ega_token.txt -rf EGAF00000406021 -re passkey -label req_EGAF00000406021

#Download
java -jar /apps/ega/download/default.jar -pf ega_token.txt -dr req_EGAF00000406021 -nt 8

Error log

Files to download in this request: 5
Start Download Process: 4 (max:15) parallel threads
Iteration 1: 5 files.
Starting download: /EGAZ00001016120/D0LA3ACXX_2.bam.gpg  (0/5)
Starting download: /EGAZ00001016120/D0LA3ACXX_2.bam.gpg  (1/5)
Starting download: /EGAZ00001016120/D0LA3ACXX_2.bam.gpg  (2/5)
Starting download: /EGAZ00001016120/D0LA3ACXX_2.bam.gpg  (3/5)
Download Active: 5 jobs submitted, 0 completed.

Output: Ticket: b1d744f6-a180-4892-948b-a7c2a8de9f49
URL: http://ega.ebi.ac.uk/ega/rest/ds/v2/downloads/b1d744f6-a180-4892-948b-a7c2a8de9f49
Establishing Data Stream for: b1d744f6-a180-4892-948b-a7c2a8de9f49
Data Stream for b1d744f6-a180-4892-948b-a7c2a8de9f49 is established
Path: /gpfs/ctgs0/fastscratch/kimh/tmp/Costello/EGAD00001000714/_EGAZ00001016120_D0LA3ACXX_2.bam.cip
Starting transfer loop for b1d744f6-a180-4892-948b-a7c2a8de9f49 (true)

Download Error occurred at Stage 4
Error: Read timed out
Completed download: /EGAZ00001016120/D0LA3ACXX_2.bam.gpg: false  (1/5)
Download Failed: /EGAZ00001016120/D0LA3ACXX_2.bam.gpg  b1d744f6-a180-4892-948b-a7c2a8de9f49
Download Failed: /EGAZ00001016120/D0LA3ACXX_2.bam.gpg  2c54b30d-369c-4738-b18b-560ecd3e1373

ADD REPLY • link 6.6 years ago by Samir ▴ 210

0

Entering edit mode

Both issues may have been better directed to the EGA HelpDesk, who, from my experience, respond rapidly. Be wary, however, that in the UK, next Monday (Monday 7th May 2018) is a public holiday.

ADD REPLY • link 6.6 years ago by Kevin Blighe 88k

0

Entering edit mode

I'm having the same problem. Any solutions? I did contact EGA help desk and they said that their client had some issues which has been fixed and is up and running. But I still seem to have this problem.

ADD REPLY • link 6.4 years ago by 29aprilkrutika ▴ 10

0

Entering edit mode

If EGA support are sure that no problems exist on their end then you may want to talk with your local IT to see if there are any firewall restrictions that are preventing you from downloading.

ADD REPLY • link 6.4 years ago by GenoMax 147k

0

Entering edit mode

I'm using nohup in an ec2 instance on AWS to download the data. Can there be any issue with that?

ADD REPLY • link 6.4 years ago by 29aprilkrutika ▴ 10

0

Entering edit mode

I am having the exact same problem so I still haven't finished downloading the files. A few weeks ago, I had been contacting with the EGA staff for at least 20 emails and they were saying that they had a server issue.

Need some solutions too!

ADD REPLY • link 6.3 years ago by miaowzai ▴ 390

score 1 · Answer 1 · 2018-08-07

1

Entering edit mode

6.3 years ago

29aprilkrutika ▴ 10

I requested EGA to create a aspera account for me with the dataset, it was easy enough to download the data using their download.sh script. You will need to ask for a decryption key for the data as well.

ADD COMMENT • link 6.3 years ago by 29aprilkrutika ▴ 10

score 1 · Answer 2 · 2018-09-11

I have been having this problem for a long time and now it is solved by using their python3 client. Try it here: https://ega-archive.org/download/downloader-quickguide-v3#downloadClient

I am not entirely sure why the same downloading works with the python3 client but not with the other methods though... (the other methods always show a server error on their side) I believe requesting an aspera downloading works too!