Question

Downloading from MG-RAST

0

Entering edit mode

10.0 years ago

clm406 • 0

I am trying to download ~100 public amplicon sets from MG RAST I am using a slightly modified script from a previous entry here on BioStars (Downloading Data From Mg-Rast) that looks like this

cat names.csv | while read line; do curl http://api.metagenomics.anl.gov/1/download/"$line"?file=100.2 >$line.fna; done

where names.csv is a list of the numbers of the files in which I am interested.

This will write all the files for the entries names.csv like it is supposed to, but each of those files only contain an error message:

<html>
<head><title>400 Bad Request</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>

I can download the files if I replace "$line" with the actual number in my browser or if I run the command

curl http://api.metagenomics.anl.gov/1/download/metagenomenumber?file=100.2 > metagenomenumber.fna

from my terminal window, but I would much prefer to not have to do each one manually.

Any ideas on why my shell script may be failing?

Thanks!

Christy

MG-RAST • 4.8k views

ADD COMMENT • link updated 2.7 years ago by Ram 44k • written 10.0 years ago by clm406 • 0

0

Entering edit mode

Try replacing $line with ${line} maybe? Just an idea since I don't have access to your CSV file.

ADD REPLY • link updated 2.7 years ago by Ram 44k • written 10.0 years ago by Dan D 7.4k

Ram · Answer 1 · 2014-12-14

0

Entering edit mode

10.0 years ago

hpmcwill ★ 1.2k

The URL of the resource to be downloaded should generally be completely enclosed in quotes to prevent any of its contents being interpreted by the shell as additional arguments or file patterns. In your case this means something like the following will work as you expect:

cat names.csv | while read line; do curl "http://api.metagenomics.anl.gov/1/download/$line?file=100.2" > "$line.fna"; done

As noted by Dan D, you might want to protect yourself further by making the variable boundaries explicit by using something like:

cat names.csv | while read line; do curl "http://api.metagenomics.anl.gov/1/download/${line}?file=100.2" > "${line}.fna"; done

Although this is unnecessary in this case.

ADD COMMENT • link updated 2.7 years ago by Ram 44k • written 10.0 years ago by hpmcwill ★ 1.2k

0

Entering edit mode

Is this question and answer about the Unix language for cURL? Is there a similar function for 'while' and 'do' for cURL in a windows command prompt? I found that 'type' is a similar command to 'cat'.

ADD REPLY • link updated 2.7 years ago by Ram 44k • written 9.7 years ago by kbrannen • 0