How To Download All Est Sequences For Organism Xx From Ncbi?

Question

[Yannick Wurm]

I want to download all EST sequences from Genbank that are in the Order Hymenoptera.

This is easy by pointing and clicking: http://www.ncbi.nlm.nih.gov/sites/entrez?term=Hymenoptera&cmd=Search&db=nucest

However, I cannot get the efetch/eutils syntax right to do this in the commandline.

Note that contrarily to the following question, I do not know the identifiers for these ESTs: How To Retrive The Dna Sequence From A List Of Embl And Geneid

Thanks!

yannick

Answer 1

[Pierre Lindenbaum]

try db=nucest, retmax=300000 and term=Hymenoptera[Organism]

http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=nucest&retmax=300000&term=Hymenoptera[Organism]

and then call efetch with the list of ID, e.g:

http://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=nuccore&id=281381068&rettype=fasta

[Pierre Lindenbaum]

or you can put more than one id with the same parameter 'id':

curl 'http://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=nuccore&id=281381068<,281381069,281381067&rettype=fasta'

[Yannick Wurm]

Merci, Pierre! Your response implies that one cannot avoid calling efetch 300,000 times? I'm a bit scared NCBI will think I'm running a denial-of-service attack!!

Also, is it possible to get eutils output in .txt rather than xml?

[Pierre Lindenbaum]

no, you can also use the parameter 'usehistory' in esearch (http://eutils.ncbi.nlm.nih.gov/entrez/query/static/esearch_help.html#History), this will give your a 'WebEnv' that you'll later use with one and only efetch query

[Yannick Wurm]

excellent, thanks!

[Yannick Wurm]

And bioruby provides a great wrapper. http://bioruby.org/rdoc/classes/Bio/NCBI/REST.html

Answer 2

[Darked89]

You can start with TaxonomyBrowser:

• http://www.ncbi.nlm.nih.gov/Taxonomy/Browser/wwwtax.cgi
• Click on checkbox "Nucleotide EST",
• Click Display button
• Click on 290,764 link
• You get there: http://bit.ly/d3zNOf
• Pull-down menus: Summary->Fasta, Send to -> File (regular download of a file in i.e. Firefox)

[Yannick Wurm]

thanks - but I was looking for "non-clicking" approach to this (see Pierre's answer)

Answer 3

[Yannick Wurm]

Following up on Pierre's answer & my comment, here's how to do it in ruby, using NCBI's eutils.

#!/sw/bin/ruby
require 'bio'


ncbi = Bio::NCBI::REST.new
Bio::NCBI.default_email = "[e-mail]"
sequenceIDs = ncbi.esearch("Hymenoptera[organism]",
                           { "db"=>"protein", "rettype"=>"gb", "retmax"=> 10000000})    
sequences   = ncbi.efetch(ids = sequenceIDs,
                          {"db"=>"protein", "rettype"=>"fasta", "retmax"=> 10000000})

# ncbi returns a single big string with records separated by two newlines              
sequences.gsub!("\n\n", "\n")
File.open('genbankHymenopteranProts.fasta', 'w') {|f| f.write(sequences +"\n") }

[Rm]

Error:

/usr/lib/ruby/gems/1.8/gems/bio-1.4.3.0001/lib/bio/io/ncbirest.rb:151:in `ncbi_check_parameters': Set email parameter for the query, or set Bio::NCBI.default_email = "(your email address)" (RuntimeError)
        from /usr/lib/ruby/gems/1.8/gems/bio-1.4.3.0001/lib/bio/io/ncbirest.rb:128:in `ncbi_post_form'
        from /usr/lib/ruby/gems/1.8/gems/bio-1.4.3.0001/lib/bio/io/ncbirest.rb:268:in `esearch'
        from /usr/lib/ruby/gems/1.8/gems/bio-1.4.3.0001/lib/bio/io/ncbirest.rb:265:in `step'
        from /usr/lib/ruby/gems/1.8/gems/bio-1.4.3.0001/lib/bio/io/ncbirest.rb:265:in `esearch'
        from eftech.ry:7

[Yannick Wurm]

Indeed you need to supply your email (I didn't want to put my own; added a blank one above):

Bio::NCBI.default_email = "your email"