Scanning for ultra-hot topics in science via pubmed, or using other tools

Question

[LauferVA]

A search in pubmed for chatGpt returns:

• 0 articles in 2021
• 4 articles in 2022
• 1238 articles (as of Sept 17) in 2023

https://pubmed.ncbi.nlm.nih.gov/?term=chatGpt&filter=datesearch.y_10&size=100

I am not aware of a more rapid growth in scholarship relating to any scientific idea or advance whatsoever, though there may be one...

My question is, how might one search for such a thing? No restriction on tools that could be used, could use EntrezUtils and BioPython, Google and webscripting, anything. Thoughts?

Answer 1

[GenoMax]

Because you are limited to 1000 results via EntrezDirect searches a live option is not going to work. And that may apply to most other technologies you mention above.

You can download the XML PubMed database files from https://ftp.ncbi.nih.gov/pubmed/baseline/ (up to end of of 2022). Or to keep things simpler just 2023 updates here https://ftp.ncbi.nih.gov/pubmed/updatefiles/

Parsing the XML files for tiles/abstracts locally should allow you to use a "word cloud" like app to find the information you need.

---

How ridiculous the live search can be is illustrated by this search for one (or two) days of publications.

$ esearch -db pubmed -query "("2023/01/09"[Date - Publication] : "2023/01/10"[Date - Publication])"

results in 10K+ results.

Looking at the title of I think just first 1K results you can get the most oft appearing words in the titles.

$ esearch -db pubmed -query "("2023/01/09"[Date - Publication] : "2023/01/10"[Date - Publication])" | efetch -format docsum | xtract -pattern DocumentSummary -element Title |  tr ' ' '\n' | sort | uniq -c | sort -k1,1nr > search_res

[LauferVA]

GenoMax , thank you. this is a really great start.

i will likely use a python based approach, but other than that, i think ill proceed exactly as suggested.

with kindness, and thanks,

VAL