Aspera ENA download: errors and permissions?

Question

[f.a.galkin]

I am trying to download 20TB of metagenomic data from ENA. However their standard ftp service hardly ever reaches 400KB/s and it would take me literally ages to download it all

I've been looking into the Aspera alternative, but am currently facing problems with it.

I've installed it and tried to download ERR1729192 run:

ascp -v -i ~/.aspera/connect/etc/asperaweb_id_dsa.openssh -k 1 -T -l 100M -L- [e-mail]/vol1/fastq/ERR172/002/ERR1729192/ERR1729192.fastq.gz ./

And I get the following log:

LOG Configuration: using v2 configuration file "/home/galkin/.aspera/connect/etc/aspera.conf", user -
LOG Initializing FASP version 3.8.1.161147, license max rate=(unlimited), account no.=1, license no.=1 product=6
LOG Configured symlink actions: create=1, follow=1, follow_wide=0, skip=0
LOG Aspera Connect version 3.8.1.161274

LOG Alternate log directory: "-"
ascp: no remote host specified
Startup failed, exit
ERR no remote host specified
ERR Missing mandatory arguments
LOG FASP Session Start uuid=cfe1b5c8-890d-40c8-a93f-466f37d0b51c op=recv status=failed errcode=106 errstr="Usage error" source= dest= source_prefix=- local=0.0.0.0:0 peer=0.0.0.0:0 tcp_port=22 os="Linux 4.4.0-62-generic #83-Ubuntu SMP We" ver=3.8.1.161147 lic=6:1:1 peeros="-" peerver=- peerlic=0:0:0 proto_sess=20003 proto_udp=20000 proto_bwmeas=20000 proto_data=2000d
LOG FASP Session Params uuid=cfe1b5c8-890d-40c8-a93f-466f37d0b51c userid=0 user="-" targetrate=10000000 minrate=0 rate_policy=fair cipher=none resume=0 create=0 ovr=0 times=0 precalc=no mf=0 mf_path=- mf_suffix=.aspera-inprogress partial_file_suffix=- files_encrypt=no files_decrypt=no file_csum=NONE dgram_sz=1492 prepostcmd=- tcp_mode=no rtt_auto=no cookie="-" vl_proto_ver=1  peer_vl_proto_ver=0 vl_local=0 vlink_remote=0 vl_sess_id=0 srcbase=- rd_sz=0 wr_sz=0 cluster_num_nodes=1 cluster_node_id=0 cluster_multi_session_threshold=-1 range=0-0 keepalive=no test_login=no proxy_ip=- net_rc_alg=alg_none exclude_older/newer_than=0/0
LOG FASP Session Stop uuid=cfe1b5c8-890d-40c8-a93f-466f37d0b51c op=recv status=failed errcode=106 errstr="Usage error" source= dest= source_prefix=- local=0.0.0.0:0 peer=0.0.0.0:0 tcp_port=22 os="Linux 4.4.0-62-generic #83-Ubuntu SMP We" ver=3.8.1.161147 lic=6:1:1 peeros="-" peerver=- peerlic=0:0:0 proto_sess=20003 proto_udp=20000 proto_bwmeas=20000 proto_data=2000d
LOG FASP Session Params uuid=cfe1b5c8-890d-40c8-a93f-466f37d0b51c userid=0 user="-" targetrate=10000000 minrate=0 rate_policy=fair cipher=none resume=0 create=0 ovr=0 times=0 precalc=no mf=0 mf_path=- mf_suffix=.aspera-inprogress partial_file_suffix=- files_encrypt=no files_decrypt=no file_csum=NONE dgram_sz=1492 prepostcmd=- tcp_mode=no rtt_auto=no cookie="-" vl_proto_ver=1  peer_vl_proto_ver=0 vl_local=0 vlink_remote=0 vl_sess_id=0 srcbase=- rd_sz=0 wr_sz=0 cluster_num_nodes=1 cluster_node_id=0 cluster_multi_session_threshold=-1 range=0-0 keepalive=no test_login=no proxy_ip=- net_rc_alg=alg_none exclude_older/newer_than=0/0
LOG FASP Session Statistics [Receiver] id=cfe1b5c8-890d-40c8-a93f-466f37d0b51c delay=0ms rex_delay=0ms ooo_delay=0ms solicited_rex=0.00% rcvd_rex=0.00% rcvd_dups=0.00% ave_xmit_rate 0.00Kbps effective=0.00% effective_rate=0.00Kbps (detail: good_blks 0 bl_total 0 bl_orig 0 bl_rex 0 dup_blks 0 dup_last_blks 0 drop_blks_xnf 0) (sndr ctl: sent 0 rcvd 0 lost 0 lost 0.00%) (rcvr ctl: sent 0 rcvd 0 lost 0 lost 0.00%) (rex  ctl: sent 0 rcvd 0 lost 0 lost 0.00%) (progress: tx_bytes 0 file_bytes 0 tx_time 0) rex_xmit_blks 0  xmit_total 0 rex_xmit_pct 0.00%

Using enaBrowserTools does not help:

enaDataGet -a -f fastq ERR1729192

Downloading file with md5 check:fasp.sra.ebi.ac.uk:/vol1/fastq/ERR172/002/ERR1729192/ERR1729192.fastq.gz
/home/galkin/.aspera/connect/bin -QT -L /home/galkin/ERR1729192/logs -l 100M -P33001  -i /home/galkin/.aspera/connect/etc/asperaweb_id_dsa.openssh [e-mail]:/vol1/fastq/ERR172/002/ERR1729192/ERR1729192.fastq.gz ./ERR1729192
/bin/sh: 1: /home/galkin/.aspera/connect/bin: Permission denied
Error with Aspera transfer: [Errno 2] No such file or directory: './ERR1729192/ERR1729192.fastq.gz'
Downloading file with md5 check:fasp.sra.ebi.ac.uk:/vol1/fastq/ERR172/002/ERR1729192/ERR1729192.fastq.gz
/home/galkin/.aspera/connect/bin -QT -L /home/galkin/ERR1729192/logs -l 100M -P33001  -i /home/galkin/.aspera/connect/etc/asperaweb_id_dsa.openssh [e-mail]:/vol1/fastq/ERR172/002/ERR1729192/ERR1729192.fastq.gz ./ERR1729192
/bin/sh: 1: /home/galkin/.aspera/connect/bin: Permission denied
Error with Aspera transfer: [Errno 2] No such file or directory: './ERR1729192/ERR1729192.fastq.gz'
Failed to download file after two attempts
Deleting directory ERR1729192
ERROR: Something unexpected went wrong please try again.
If problem persists, please contact [e-mail] for assistance.

I have contacted our administrator and he says that our firewall settings are ok.

Meanwhile, when I am using this example I have found on the Internet, everything works fine and I can download at 90MB/s

ascp -QTr -l 400M -i ~/.aspera/connect/etc/asperaweb_id_dsa.openssh  [e-mail]:/sra/sra-instant/reads/ByStudy/sra/SRP/SRP009/SRP009247 .

Has anybody faced such a problem? What permission am I lacking? Is that a problem on my server side, or I need to ask for some kind of permission from ENA to use Aspera? What does "no remote host specified" mean?

[Devon Ryan]

You're missing a colon after the server name: [e-mail]:/vol1/fastq/ERR172/002/ERR1729192/ERR1729192.fastq.gz

[smileszh]

the link is err. ascp need special link. dot using copy form NCPBI !

Answer 1

[ATpoint]

You may try following the steps of my recent tutorial.

In your case there are two things:

1) You use the wrong path. Rather than fasp.sra.ebi.ac.uk is must be [e-mail] and

2) there must be a ":" after .uk, so [e-mail]:/vol1(...)

This command worked for me

ascp -QT -l 300m -P33001 -i "$HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh" [e-mail]:/vol1/fastq/ERR172/002/ERR1729192/ERR1729192.fastq.gz ./

Edit 07/2022:

If anyone stumbles over this post and needs to set up Aspera, here is a 2022 working solution: Setting up Aspera Connect (ascp) on Linux and macOS

[f.a.galkin]

Thank you for the quick reply. I will make sure to study your tutorial asap. But have you used enaBrowserTools? It is supposed to take in only a run accession and look up all the needed links by itself, and I still fail at downloading

[ATpoint]

No, I never used this one. Always downloaded via command line as described in the tutorial. I also modified my answer.

[f.a.galkin]

Wow, thanks a lot Your line works absolutely fine for me too. All that's left is to put up a script together

[ATpoint]

You're very welcome. If the answer helped you, please consider to mark it as accepted to help others in the future ;-)