Maximum number of efetch

Question

[horsedog]

Hi,

I'm retrieving protein sequences from UID using efetch. I got more than 30000 UID but after running python script, it only gave me exact 10000 sequences, I'm wondering is there any limitation of number of retrieving, like 10000? Is it possible to get more 30000 at one time? Thanks

[GenoMax]

In addition to @Pierre's note below consider this. Time mentioned below is US East coast time.

In order not to overload the E-utility servers, NCBI recommends that users post no more than three URL requests per second and limit large jobs to either weekends or between 9:00 PM and 5:00 AM Eastern time during weekdays. Failure to comply with this policy may result in an IP address being blocked from accessing NCBI. If NCBI blocks an IP address, service will not be restored unless the developers of the software accessing the E-utilities register values of the tool and email parameters with NCBI.

[GenoMax]

You can probably retrieve sequences much more efficiently using blastdbcmd from blast+ suite and a local copy of nr blast database.

Answer 1

[Pierre Lindenbaum]

https://www.ncbi.nlm.nih.gov/books/NBK25499/

Total number of UIDs from the retrieved set to be shown in the XML output (default=20).

Increasing retmax allows more of the retrieved UIDs to be included in the XML output, up to a maximum of 100,000 records.

To retrieve more than 100,000 UIDs, submit multiple esearch requests while incrementing the value of retstart

[horsedog]

Hi Pierre, is it possible to just get the fasta file instead of XML file? And I have no idea where to add retmax or retstart, this is my code: from Bio import Entrez Entrez.email = "[e-mail]" blast = open("file.txt").read() handle = Entrez.efetch(db="protein", id= blast, rettype="fasta") print(handle.read())

[yarmda]

Hi Pierre, do you know if looping through the records returned maintains an open ftp connection to NCBI? I have a firewall that doesn't allow ftps connections to remain open for long and my loop fails somewhere between 3 and 10 iterations. I suspect this is due to the ftp connection. I don't believe these iterations could hit the 3 requests per second maximum.