How to fix error in Entrez-Direct: 429 Too Many Requests

Question

[f-rasmussen]

I am using Entrez-direct to convert rs-ID to HGNC-symbol, which I need for downstream analysis.

more id

rs3828049
rs10907177
rs9442400
rs6687776
rs11579015
rs6671356
rs12757754
rs2298217
rs9442380
rs1891905

I then use the following code to obtain the HGNC gene symbols for each SNP rs-ID. There are about 500,000 rs id's in "id".

for i in `cat ./id`; do esearch -db snp -query "${i} AND human [orgn]" | esummary | xtract -pattern DocumentSummary -element SNP_ID,NAME |  uniq >> rs_hgnc.txt ; done

I then get the following error message for some of the id's, resulting in a blank HGNC-symbol. The script keeps running in spite of this message, but I want to fix it since I want hgnc-symbols for ALL of my SNPs. Does anyone know how to get around this problem?

The error message is displayed below:

429 Too Many Requests
No do_post output returned from 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=snp&query_key=1&WebEnv=MCID_606af2ce96bf9a5edc2dce36&retstart=0&retmax=3&version=2.0&edirect=7.40&tool=edirect&email=user@mail+'
Result of do_post http request is
$VAR1 = bless( {
                 '_rc' => 429,
                 '_content' => '{"error":"API rate limit exceeded","api-key":"195.215.246.170","count":"4","limit":"3"}
',
                 '_protocol' => 'HTTP/1.1',
                 '_request' => bless( {
                                        '_method' => 'POST',
                                        '_uri' => bless( do{\(my $o = 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi')}, 'URI::https' ),
                                        '_headers' => bless( {
                                                               'user-agent' => 'libwww-perl/6.31',
                                                               '::std_case' => {
                                                                                 'if-ssl-cert-subject' => 'If-SSL-Cert-Subject'
                                                                               },
                                                               'content-type' => 'application/x-www-form-urlencoded'
                                                             }, 'HTTP::Headers' ),
                                        '_uri_canonical' => $VAR1->{'_request'}{'_uri'},
                                        '_content' => 'db=snp&query_key=1&WebEnv=MCID_606af2ce96bf9a5edc2dce36&retstart=0&retmax=3&version=2.0&edirect=7.40&tool=edirect&email=user@mail+'
                                      }, 'HTTP::Request' ),
                 '_headers' => bless( {
                                        'x-xss-protection' => '1; mode=block',
                                        'retry-after' => '2',
                                        'x-ratelimit-limit' => '3',
                                        'access-control-expose-headers' => 'X-RateLimit-Limit,X-RateLimit-Remaining,Retry-After',
                                        'client-ssl-cipher' => 'ECDHE-RSA-AES256-GCM-SHA384',
                                        'client-peer' => '130.14.29.110:443',
                                        'x-ua-compatible' => 'IE=Edge',
                                        'client-ssl-cert-subject' => '/C=US/ST=Maryland/L=Bethesda/O=National Library of Medicine/OU=National Center for Biotechnology Information/CN=*.ncbi.nlm.nih.gov',
                                        'connection' => 'close',
                                        'server' => 'Finatra',
                                        'client-ssl-socket-class' => 'IO::Socket::SSL',
                                        'x-ratelimit-remaining' => '0',
                                        'strict-transport-security' => 'max-age=31536000; includeSubDomains; preload',
                                        'vary' => 'Accept-Encoding',
                                        'date' => 'Mon, 05 Apr 2021 11:21:50 GMT',
                                        'content-length' => '88',
                                        'client-response-num' => 1,
                                        'content-type' => 'application/json',
                                        'client-date' => 'Mon, 05 Apr 2021 11:21:51 GMT',
                                        '::std_case' => {
                                                          'client-response-num' => 'Client-Response-Num',
                                                          'client-peer' => 'Client-Peer',
                                                          'x-ua-compatible' => 'X-UA-Compatible',
                                                          'client-date' => 'Client-Date',
                                                          'content-security-policy' => 'Content-Security-Policy',
                                                          'client-ssl-cert-subject' => 'Client-SSL-Cert-Subject',
                                                          'client-ssl-cert-issuer' => 'Client-SSL-Cert-Issuer',
                                                          'x-xss-protection' => 'X-XSS-Protection',
                                                          'client-ssl-socket-class' => 'Client-SSL-Socket-Class',
                                                          'access-control-expose-headers' => 'Access-Control-Expose-Headers',
                                                          'x-ratelimit-remaining' => 'X-RateLimit-Remaining',
                                                          'x-ratelimit-limit' => 'X-RateLimit-Limit',
                                                          'strict-transport-security' => 'Strict-Transport-Security',
                                                          'client-ssl-cipher' => 'Client-SSL-Cipher'
                                                        },
                                        'content-security-policy' => 'upgrade-insecure-requests',
                                        'client-ssl-cert-issuer' => '/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA'
                                      }, 'HTTP::Headers' ),
                 '_msg' => 'Too Many Requests'
               }, 'HTTP::Response' );

Answer 1

[GenoMax]

Have you signed up for NCBI API KEY? If not do that first. Even then you need to remain cognizant of limits per unit time. Manage your queries by adding wait/sleep steps.

Answer 2

[Asaf]

Thousands is way too many. There are ways to overcome this, one is using search history: https://www.ncbi.nlm.nih.gov/books/NBK25498/#chapter3.Application_3_Retrieving_large Another is by obtaining an API key and setting the environment variable NCBI_API_KEY to that value which will allow you move requests but it won't scale up to thousands. You can add a sleep command after each fetch which might help but the best way would be to find this data in their ftp server and download the entire table.