Question

Download of FASTQ files from the European Nucleotide Archive (ENA) using Aspera

1

Entering edit mode

5.1 years ago

r.castellan ▴ 10

Hi there, I am trying to retrieve sequencing data from ENA using the Aspera client and following the tutorial:

Fast download of FASTQ files from the European Nucleotide (ENA)

The tutorial has been really helpful and I followed each step. I get a download.txt file which is a list of items looking like:

ascp -QT -l 300m -P33001 -i $HOME/Applications/Aspera Connect.app/Contents/Resources/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/SRR898/002/SRR8989702/SRR8989702_1.fastq.gz .

I then followed the tutorial and ran:

while read LIST; do
$LIST; done < download.txt

This leads to either two errors

Startup failed, exit
ascp: no remote host specified

Or:

-bash: $LIST: ambiguous redirect

I also tried running:

$HOME/Applications/Aspera Connect.app/Contents/Resources/asperaweb_id_dsa.openssh

And get a permission denied error.

I'm new to this and haven't found a solution yet.

RNA-Seq sequencing • 5.6k views

ADD COMMENT • link updated 3.0 years ago by ATpoint 86k • written 5.1 years ago by r.castellan ▴ 10

0

Entering edit mode

This leads to either two errors

What are you doing differently to get the two errors?

ADD REPLY • link 5.1 years ago by manuel.belmadani ★ 1.4k

0

Entering edit mode

If I hit return after do I tend to get the second error and if I don't the first one. Thanks!

ADD REPLY • link 5.1 years ago by r.castellan ▴ 10

0

Entering edit mode

I am getting following error while downloading fastq files with ascp

(base) [nikhil@hpc04 dmd_SRA]$ ascp -QT  -l 300m -P33001 -i ~/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:vol1/fastq/ERR353/008/ERR3537668/ERR3537668_1.fastq.gz ./

ascp: Target address not available 
Startup failed, exit

Please guide me in this regard

ADD REPLY • link updated 3.2 years ago by ATpoint 86k • written 3.2 years ago by nikhilshinde0909 • 0

0

Entering edit mode

The command runs smoothly on my machine. Can you try:

ascp -QT -l 300m -P33001 -i $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:vol1/fastq/ERR353/008/ERR3537668/ERR3537668_1.fastq.gz .

ADD REPLY • link 3.2 years ago by ATpoint 86k

0

Entering edit mode

Just wanted to mention that the problem persists even after specifying the home directory with $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh.

ADD REPLY • link 3.0 years ago by sbstevenlee ▴ 480

0

Entering edit mode

Hello, did you ever figure out what caused the issue? I encountered the exact same issue while running on server (let's call this server A):

(fuc) [sbslee@bdcm03 ~]$ .aspera/connect/bin/ascp -QT -l 100m -P33001 -k 1 -i .aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:vol1/run/ERR323/ERR3239277/NA06986.final.cram .
ascp: Target address not available 
Startup failed, exit

The funny thing is, the exact same command works just fine in my local Linux PC:

(fuc) [sbslee@localhost Desktop]$ .aspera/connect/bin/ascp -QT -l 100m -P33001 -k 1 -i .aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:vol1/run/ERR323/ERR3239277/NA06986.final.cram .
NA06986.final.cram                                                                                                                      10% 1539MB  1.3Mb/s    38:04 ETA

So I was thinking it might be a server-related issue, but I can't think of any reason why. I also confirmed that in a different server (call it server B), the command works OK. So it got to be an issue in server A, but I didn't change any configuration, etc., so I'm not sure why I'm seeing this problem. Server A is connected to Internet for sure (I can download stuff such as $ git clone). Any help would be much appreciated!

ADD REPLY • link 3.0 years ago by sbstevenlee ▴ 480

0

Entering edit mode

I would talk to the server admins and discuss potential problems. It is obviously a specific problem with connection on that particilar server. Maybe some kind of firewall that is in place or ports that are not open, idk.

ADD REPLY • link 3.0 years ago by ATpoint 86k

score 1 · Answer 1 · 2019-11-20

1

Entering edit mode

5.1 years ago

ATpoint 86k

Thanks for reporting the issue. I could reproduce the error and think it is due to the whitespace in the name of the Aspera folder. Could you try to rename Aspera Connect into something without whitespace e.g. AsperaConnect.app, then adjust the download command accordingly and try again. I guess that is Mac-specific because (wisely) Linux does not use a whitespace in the folder name. I Did it on my machine and the errors disappeared, still I got a time-out message. Not sure if this is due to an error at ENA which happens at times, so if you cannot download, maybe wait a day and try again. Connection errors are rather frequent in my experience. I will double check when I've access to my workstation tomorrow.

==> Edit: The whitespacing is not an issue if properly escaped. I updated the command in the tutorial so that the awk command does not mess it up. It should be fine towards that initial error you got.

ADD COMMENT • link 5.1 years ago by ATpoint 86k

0

Entering edit mode

Thanks a lot, will try that now and update.

ADD REPLY • link 5.1 years ago by r.castellan ▴ 10

0

Entering edit mode

Just checked on both Linux and Mac and the same problem, I think this is due to server problems at ENA. I will update the tutorial towards this whitespace problem, thanks again for reporting.

ADD REPLY • link 5.1 years ago by ATpoint 86k

0

Entering edit mode

Oh right! Sorry only saw your reply now. No worries, thanks for helping

ADD REPLY • link 5.1 years ago by r.castellan ▴ 10

0

Entering edit mode

So I changed it and I get a new error:

ascp: Private key file not found at path /Users/Raphael/.ssh/$HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh

I'm novice, not sure it was worth pointing out, but does this path look strange?

I also ran the following:

$HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh

I get the permission denied error still

ADD REPLY • link 5.1 years ago by r.castellan ▴ 10

0

Entering edit mode

Is there a file at the location? What do you get if you do ls -l $HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh? If your SSH key is in some other location then you will need to use that path.

Are you running macOS catalina BTW?

ADD REPLY • link 5.1 years ago by GenoMax 148k

0

Entering edit mode

I get: -rw-rw-r--@ 1 Raphael staff 668 8 Nov 15:12 /Users/Raphael/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh

I am running Mojave still

ADD REPLY • link 5.1 years ago by r.castellan ▴ 10

0

Entering edit mode

It is normal that you get the permission denied when calling this file, it is also not an executable so no point in trying to run it. Your problem is here:

/Users/Raphael/.ssh/$HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh

Remove /Users/Raphael/.ssh/ from it as $HOME is already there.

Try and it should work unless there is something odd going on with your environment:

ascp -QT -l 300m -P33001 -i $HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/SRR898/002/SRR8989702/SRR8989702_1.fastq.gz .

ADD REPLY • link 5.1 years ago by ATpoint 86k

0

Entering edit mode

So when I ran the code by itself it worked. However, it fails when I run:

while read LIST; do
$LIST; done < download.txt

when I head download.txt I get:

ascp -QT -l 300m -P33001 -i $HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/SRR898/002/SRR8989702/SRR8989702_1.fastq.gz .

ADD REPLY • link 5.1 years ago by r.castellan ▴ 10

0

Entering edit mode

What is the error mesage?

ADD REPLY • link 5.1 years ago by ATpoint 86k

0

Entering edit mode

sorry I had to wait 6 hours as I had put up 5 posts...

the error message is the same as before:

ascp: Private key file not found at path /Users/Raphael/.ssh/$HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh

ADD REPLY • link 5.1 years ago by r.castellan ▴ 10

1

Entering edit mode

It is puzzling that your $HOME variable is expanding correctly to /Users/Raphael when you use it here ls -l $HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh but when you try to run the loop it seems to change to something strange /Users/Raphael/.ssh/$HOME and does not properly expand.

What do you get when you echo $HOME?

What happens if you simply try one file like this? Does this work?

ascp -QT -l 300m -P33001 -i /Users/Raphael/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/SRR898/002/SRR8989702/SRR8989702_1.fastq.gz .

ADD REPLY • link 5.1 years ago by GenoMax 148k

0

Entering edit mode

So echo $HOME returns:

/Users/Raphael

The following worked, so I guess it's a problem in the loop which adds the extra bits?

ascp -QT -l 300m -P33001 -i /Users/Raphael/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/SRR898/002/SRR8989702/SRR8989702_1.fastq.gz .

ADD REPLY • link 5.1 years ago by r.castellan ▴ 10

1

Entering edit mode

It is indeed odd that $HOME seems to improperly expand. I used $HOME in the tutorial commands to provide a generic command, but glad it works now. You can simply edit the awk command that created the download.txt by replaceing the $HOME with /Users/Raphael/. This should be independent of what the loop does as it only calls what is inside the download list.

ADD REPLY • link 5.1 years ago by ATpoint 86k

0

Entering edit mode

Great! Thanks so much for your help!

ADD REPLY • link 5.1 years ago by r.castellan ▴ 10

0

Entering edit mode

It's running nicely, the session is timing out every now and then on certain files, is there a way to automatically determine which ones have failed and re-run them? Maybe matching the output files with the download.txt file?

ADD REPLY • link 5.1 years ago by r.castellan ▴ 10