Fetch table from clinvar database according to a list of rsid

Question

[ashaneev07]

Hi

I have a list of rsid and i want to search against clinvar database and print the condition_germline column with respect to each rsid. Anyway, i have got a script.

use strict;
use warnings;
use LWP::Simple;
use HTML::TableExtract;

# Read list of rsids from file
my $rsids_file = 'rsids.txt';
open(my $fh, '<', $rsids_file) or die "Can't open $rsids_file: $!";
my @rsids = <$fh>;
close($fh);

# Loop through each rsid
foreach my $rsid (@rsids) {
    chomp($rsid);

    # Construct URL for ClinVar search
    my $url = "https://www.ncbi.nlm.nih.gov/clinvar/variation/$rsid/";

    # Fetch web content
    my $content = get($url);
    unless (defined $content) {
        warn "Couldn't get $url: ", $!;
        next;
    }

    # Extract table
    my $te = HTML::TableExtract->new(headers => ["Condition_Germline"]);
    $te->parse($content);

    # Print condition_germline column
    foreach my $ts ($te->tables) {
        foreach my $row ($ts->rows) {
            print join("\t", @$row), "\n";
        }
    }
}

But, when its runs getting the following error.

Couldn't get https://www.ncbi.nlm.nih.gov/clinvar/variation/rs11203366/:  at .\fetch_condition.pl line 22.
Couldn't get https://www.ncbi.nlm.nih.gov/clinvar/variation/rs11203367/:  at .\fetch_condition.pl line 22.
Couldn't get https://www.ncbi.nlm.nih.gov/clinvar/variation/rs874881/:  at .\fetch_condition.pl line 22.
Couldn't get https://www.ncbi.nlm.nih.gov/clinvar/variation/rs776453694/:  at .\fetch_condition.pl line 22.
Couldn't get https://www.ncbi.nlm.nih.gov/clinvar/variation/rs80324279/:  at .\fetch_condition.pl line 22.
Couldn't get https://www.ncbi.nlm.nih.gov/clinvar/variation/rs324420/:  at .\fetch_condition.pl line 22.
Couldn't get https://www.ncbi.nlm.nih.gov/clinvar/variation/rs112766203/:  at .\fetch_condition.pl line 22.

I appreciate your suggestions.

Thank you.

[Pierre Lindenbaum]

Don't forget to follow up on your threads. If an answer was helpful, you should upvote it; if the answer resolved your question, you should mark it as accepted. You can accept more than one answer if they all work. If an answer was not really helpful or did not work, provide detailed feedback so others know not to use that answer.

• Minimum requirements for a laptop to do rna seq analysis
• Exonerate installation on server
• Error while running gatk
• Augustus and SNAP training in Maker-p pipeline
• Retrieving sequence from NCBI
• Snoscan#error#make: Nothing to be done for 'all'
• Maker Annotation pipeline Output error
• E: Package 'libbamtools-dev' has no installation candidate
• Increase memory limit in SPAdes-3.6.1
• SNP filtering from specific regions

[ashaneev07]

Hii... i have updated the script in python..But, still getting no data found. Actually the data is there, i have print the parsed html file but not print the conditions_germline = soup.find('Conditions-Germline'). Is there problem with this line?? I have attached the script with this. Kindly have a look into this.Thank you.

>     import requests
>     from bs4 import BeautifulSoup
>     
>     
>     def search_clinvar_by_rsid(rsid):
>         url = f"https://www.ncbi.nlm.nih.gov/clinvar/?term={rsid}"
>         try:
>             response = requests.get(url)
>             if response.status_code == 200:
>                 soup = BeautifulSoup(response.content, 'html.parser')
>                 #print(soup)
>                 conditions_germline = soup.find('Conditions-Germline')
>                 print(conditions_germline)
>                 if conditions_germline:
>                     conditions_text = conditions_germline.text.strip()
>                     print(f"conditions_germline data for rsid '{rsid}':")
>                     print(conditions_text)
>                 else:
>                     print(f"No conditions_germline data found for rsid '{rsid}'")
>             else:
>                 print(f"Failed to retrieve search results for rsid '{rsid}'. Status code:", response.status_code)
>         except requests.RequestException as e:
>             print("Error:", e)
>     
>     
>     def read_rsids_from_file(filename):
>         with open(filename, 'r') as file:
>             return [line.strip() for line in file]
>     
>     rsids = read_rsids_from_file('rsids.txt')
>     for rsid in rsids:
>         search_clinvar_by_rsid(rsid)

Answer 1

[Pierre Lindenbaum]

$ cat input.rs

rs11203366
rs11203367
rs874881
rs776453694
rs80324279
rs324420
rs112766203

 wget -qO - "https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/clinvar.vcf.gz" |\
   bcftools query -i "`cat input.rs  | sed 's/^rs//' | awk '{printf("%sRS=\x27%s\x27",(NR==1?"":" || "),$1);}'`" -f '%CHROM %POS %REF %ALT rs%RS %CLNDN\n'

1  17331039  G  A  rs11203366   Rheumatoid_arthritis|Abnormal_pulmonary_interstitial_morphology|PADI4-related_condition
1  17331121  T  C  rs11203367   Rheumatoid_arthritis|Abnormal_pulmonary_interstitial_morphology|PADI4-related_condition
1  17334004  G  C  rs874881     Rheumatoid_arthritis|Abnormal_pulmonary_interstitial_morphology|PADI4-related_condition
1  21838914  C  T  rs776453694  Schwartz-Jampel_syndrome_type_1|Inborn_genetic_diseases|Schwartz-Jampel_syndrome|Lethal_Kniest-like_syndrome
1  33013330  G  C  rs80324279   Reticular_dysgenesis
1  46405089  C  A  rs324420     FAAH-related_condition|FAAH_POLYMORPHISM|Polysubstance_abuse,_susceptibility_to
1  97305279  G  A  rs112766203  not_provided|Dihydropyrimidine_dehydrogenase_deficiency

[Pierre Lindenbaum]

ah ! and your code doesn't work because clinvar ID is not a rs-ID.

[ashaneev07]

so which one should i prefer? i don't need the entire results i need the condition-Germline table only. (for eg:-from, https://www.ncbi.nlm.nih.gov/clinvar/variation/294920/ need the condition-Germline table ) when i manually check with the rsid i got the result. I'm confused with this.Could you please make a clarification regarding this?