How to download lakhs of abstracts in Batch for text mining study

Question

[rohitsatyam102]

Hi everyone!!

I am new to text mining studies carried out in bioinformatics. I was learning how to use pubmed.mineR package for text mining using mulit-abstract txt file. I read a lot on RISmed package and easypubmed R packages but it seems they have some disadvantage when you have to use their output in pubmed.mineR.

pubmed.mineR uses a text file containing multiple abstracts in "abstract" format of Pubmed (one format abovet many other pubmed formats such as xml). easyPubmed have such function that can be exploited while RISmed doesn't it seems. On the other hand you can retrieve only 5000 abstracts at a time using easyPubmed while RISmed has no such upper limit.

I randomly chose a cancer type "oral cancer" and it had above 1 lakh PMIDs. RISmed successfully retrieved the abstracts(1.1 GB data) however it's output is incompatible for pubmed.mineR. On the other hand, though easypubmed had a compatible output, it has retrieval limit since it use PubMed API at the backend.

Is there a way using CLI to retrieve all ~1 lakh of abstracts in "abstract" format from pubmed since the website itself after June 2020 update has set a limit to 10,000 abstracts at a time. Here I attach a short code I used with easypubmed

library("easyPubMed")
search_topic <- 'oral cancer'
my_entrez_id <- get_pubmed_ids(search_topic)
my_entrez_id$Count
?fetch_pubmed_data
my_abstracts_txt <- fetch_pubmed_data(my_entrez_id, retmax = 142000, format = "abstract")
writeLines(my_abstracts_txt, con = "oral_cancer.txt")

Answer 1

[GenoMax]

You can use Entrezdirect (example output truncated). Be sure to sign-up for and use NCBI_API_KEY.

$ esearch -db pubmed -query "Oral cancer" | efetch -format abstract >> some_file

1. Am J Otolaryngol. 2020 Aug 16;41(6):102685. doi: 10.1016/j.amjoto.2020.102685. [Epub ahead of print]

   HPV vaccination practices and attitudes among primary care physicians since FDA approval to age 45.

   Petrusek J(1), Thorpe E(2), Britt CJ(2).

   Author information: (1)Department of Otolaryngology, Loyola University Medical Center, Maywood, IL 60153, United States of America. Electronic address: [e-mail]. (2)Department of Otolaryngology, Loyola University Medical Center, Maywood, IL 60153, United States of America.

   PURPOSE: The aim of this study was to examine HPV vaccine administration practices since FDA approval to age 45 and assess knowledge regarding HPV and its association with oropharyngeal cancer. METHODS: A survey was distributed to 86 primary care physicians at Loyola University Medical Center. The survey contained 11 questions designed to capture HPV vaccination practices, knowledge of FDA approval, and barriers to vaccination. RESULTS: 46 (53%) physicians completed the survey and 45 responses were included. Among respondents who treat males ages 9-21 and females ages 9-26, the

[rohitsatyam102]

Wow, this is such an effortless way of doing that. However, I noticed that the numbering of the abstract starts from 1 again after every 100 abstracts. Hope that won't cause any trouble during the analysis though!!.

Thanks a lot for the solution.

[rohitsatyam102]

(example output truncated).

I changed the

> some_file

to

>> somefile

Appending helps I think. Is that truncation you are referring to?

[GenoMax]

I was referring to truncation of the example. I have updated my answer to append data to the file.

[rohitsatyam102]

Wonderful. Thanks for your time. I really appreciate it. I will start looking out of R from now for solutions

[rohitsatyam102]

Be sure to sign-up for and use NCBI_API_KEY.

I am using entrezdirect package in conda. It seems not to ask for any such key.

Checking how many abstracts were fetched

grep 'PMID' abstract.txt | wc -l

[GenoMax]

It won't ask. NCBI will start throttling your queries based on description in the link I provided above, if you don't use the key.

[rohitsatyam102]

I can not access this paper. Can you share the author's copy if available? My email ID is [e-mail]

[GenoMax]

[Epub ahead of print]

That paper has not been published yet. You will find that journals will pre-send paper listings to PubMed before actual publication.

Answer 2

[rohitsatyam102]

Sharing some initial sources that I referred so that it stays all at one place Sources Referred

1. https://datascienceplus.com/search-pubmed-with-rismed/

2. https://amunategui.github.io/pubmed-query/

3. A: Full Text Retrieval From Pubmedcentral

4. Text mining of pubmed abstracts