Question

Tool:Python script to query GeneCards to get EntrezID, symbol from Ensembl geneID

2

Entering edit mode

3.3 years ago

Shred ★ 1.5k

UPDATE: Genecards recently introduced an explicit policy asking to do not use automated scraping tool. While the provided solution will (probably) still work for a single entry, multiple queries may be blocked and represents a policy violation.

Hi guys, GeneCards is one of the most comprehensive repository for gene info. While doing RNA-seq analysis, the common task to convert Ensembl geneID to Entrez ID with tools like BiomaRt may leave lots of genes without a corresponding Entrez number.

Although much of the losses are from pseudogenes or very poor characterized genes, someone may be interested to evaluate this loss, expecially if some of the lost genes populate the DEG list.

I've written this quick script in Python to query GeneCards to retain info about these uncharacterized genes. Please keep in mind that this must not be intended as a replacement for BiomaRt or other tools: GeneCards implements a protection against automatic queries and so would be impossible to assign correspondant ID to every gene in your analysis. I've made some tests with list of 150-200 genes and it seems to be ok. If you keep displaying the same error, consider to run a smaller gene list.

It can be run in a gene list mode, --list, where a file composed by 1 Ensembl geneID per line will be submitted, or in a single gene mode, --gene, passing a single ensembl geneID.

I'm working to bypass the website protection while minimizing the impact on users. Any suggestion are welcome.

Code -->

Requires BeautifulSoup

Ensembl biomart annotation • 2.9k views

ADD COMMENT • link 5 months ago by Shred ★ 1.5k

score 0 · Answer 1 · 2024-05-27

Here's a slightly changed version of this script.

I changed line 42 from 'title':'NCBI Entrez Gene' to 'title':'NCBI Gene' ).

And line 76 from res_dict = fill_dict(gene_list=args.input) to res_dict = fill_dict(gene_list=args.list)

Here's an example of how it works:

python3 get_from_gene_cards.py --list ensembl_ids.txt

Where ensembl_ids.txt looks like this:

ENSG00000018607
ENSG00000042304
ENSG00000064489

.

#!/usr/bin/python3

from bs4 import BeautifulSoup
import requests
import html
import time
import argparse

'''
Made in a boring day by @danilotat. Enjoy
'''
def get_symbol(html_file, ensgid):
    soup = BeautifulSoup(html_file, 'html.parser')
    try:
        title_tag = soup.find('title')
        if title_tag and title_tag.contents:
            symbol_tag = title_tag.contents[0]
            symbol = symbol_tag.split(' ')[0]
            if symbol == ensgid:
                return 'NaN'
            else:
                return symbol
        else:
            return 'NaN'
    except AttributeError:
        print('Genecards is blocking your requests. Please try again later with less geneid')
        exit()

def gene_card_request(ensgid):
    ''' 
    Passing Ensembl gene id, return its webpage on Genecards
    '''
    url_to_request = 'https://www.genecards.org/cgi-bin/carddisp.pl?gene=' + ensgid
    headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
    r = requests.get(url_to_request, headers=headers)
    gene_card_html = html.unescape(r.text)
    return gene_card_html

def get_entrez(html_file):
    ''' 
    Given a valid Ensembl gene id, returns Entrez gene id
    '''
    soup = BeautifulSoup(html_file, 'html.parser')
    entrez_href = soup.find_all('a', {'title': 'NCBI Gene'})
    # usually it will return redundant refer: we'll take just the first of them
    try:
        link = entrez_href[0]
        ncbi_url = link.get('href')
        entrez_id = ncbi_url.split('/')[-1]
        return entrez_id
    except IndexError:
        # empty list: no Entrez ID 
        return "NaN"

def fill_dict(gene_list):
    ''' Fill a dict with Ensembl geneid, entrez, symbol
    '''
    conv_dict = {}
    with open(gene_list, 'r') as engid_list:
        for line in engid_list:
            ensgid = line.rstrip()
            gcard_page = gene_card_request(ensgid)
            entrez_id = get_entrez(gcard_page)
            symbol = get_symbol(gcard_page, ensgid)
            conv_dict.setdefault(ensgid, []).extend([entrez_id, symbol])
            time.sleep(5)
    return conv_dict

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description="Giving a list of valid Ensembl gene ID, query GeneCards to get Entrez and symbols.")
    parser.add_argument("--list", help="Input list of Ensembl gene ID, one per line.")
    parser.add_argument("--gene", help="Single gene mode")
    args = parser.parse_args()

    if args.list is None and args.gene is None:
        parser.print_help()
        exit()

    if args.gene is None:
        res_dict = fill_dict(gene_list=args.list)
        for ensgid in res_dict.keys():
            print('{},{},{}'.format(ensgid, res_dict[ensgid][0], res_dict[ensgid][1]))
    else:
        gcard_page = gene_card_request(args.gene)
        print('{},{},{}'.format(args.gene, get_entrez(gcard_page), get_symbol(gcard_page, args.gene)))