Ensembl FTP Mirror in USA? Slow Downloads

Question

[QVINTVS_FABIVS_MAXIMVS]

Hello I work on a fairly good cluster at an R1 university in the USA. I am downloading the VEP 101 indexed cache files and it's slow as Christmas. I mean AOL speeds slow.

If I download a similar size file from AWS or NCBI it's extremely fast.

Is there a mirror for the USA containing the VEP cache files. I would only need hg38 merged VEP 101.

10 hours to download a 17GB file at 268KB/s

Give me a break, I get you Europeans don't like the USA but come on guys ;)

Answer 1

[GenoMax]

Perhaps the problem is local? Is your firewall stepping in? I just tried the download.

$ curl -O ftp://ftp.ensembl.org/pub/release-101/variation/indexed_vep_cache/homo_sapiens_vep_101_GRCh38.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 12 14.2G   12 1882M    0     0  32.4M      0  0:07:30  0:00:58  0:06:32 33.9M

[QVINTVS_FABIVS_MAXIMVS]

$ curl -O ftp://ftp.ensembl.org/pub/release-101/variation/indexed_vep_cache/homo_sapiens_merged_vep_101_GRCh38.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 16.4G    0 13.8M    0     0   733k      0  6:31:51  0:00:19  6:31:32  690k

Not sure, I am on a dedicated data mover node and can obtain 10x speeds on NCBI FTPs and AWS buckets. So I am not sure why it's painfully slow for Ensembl. Thanks!

[GenoMax]

AFAIK Ensembl ftp site has no mirrors (unlike main Ensembl site which does). May want to check with your local networking folks to see if they can shed some light. On east coast no problems.

[QVINTVS_FABIVS_MAXIMVS]

Thanks I submitted a ticket. Hopefully they will clear it up.

[osmith]

Ever get a response? Having the same issue.

[GenoMax]

Not seeing a problem.

$ curl -O ftp://ftp.ensembl.org/pub/release-101/variation/indexed_vep_cache/homo_sapiens_vep_101_GRCh38.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  3 14.2G    3  532M    0     0  21.5M      0  0:11:17  0:00:24  0:10:53 12.3M

[trodriguez2011]

May depend on time of day. I'm getting abysmal speeds on local, remote server, and cloud in eastern and western USA.

[jessime.kirk]

Just to add another example to the pile:

❯ networkQuality
==== SUMMARY ====
Upload capacity: 31.633 Mbps
Download capacity: 494.663 Mbps
Upload flows: 12
Download flows: 20
Responsiveness: Medium (216 RPM)

My overall network quality, versus trying to grab the VEP cache:

❯ curl -O https://ftp.ensembl.org/pub/release-108/variation/indexed_vep_cache/homo_sapiens_vep_108_GRCh38.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  1 21.2G    1  283M    0     0   406k      0 15:13:50  0:11:52 15:01:58  405k