Hi, thanks for all the useful work. I'm writing to ask about an issue I've recently encountered with clusterProfiler in its latest version. I'm trying to implement an approach using parallel::mclapply and some clusterProfiler functions (e.g. groupGO, enrichGO, gseGO) to process few sets of genes of interest simultaneously. When trying to perform the analyses in parallel, I'm consistently getting this kind of errors in these functions:
Ensembl site unresponsive, trying uswest mirror
Ensembl site unresponsive, trying asia mirror
Ensembl site unresponsive, trying uswest mirror
Ensembl site unresponsive, trying asia mirror
Ensembl site unresponsive, trying uswest mirror
When using the functions in a non-parallel fashion (and outside Rscript) the functions seem to work as expected... Does anyone have any experience with clusterProfiler being executed in parallel? Can be used it offline or with local databases somehow?
Any comment would be much appreciated.
Thanks for your time
There is a chance that there was a genuine outage. Ensembl sites do go down periodically.
That said, many public sites limit the number of queries you are allowed to do per unit time based on your IP address. These are public resources and they want to ensure that the resource remains accessible. You must be hitting that limit by running in parallel.
Thanks for your answer. Those were my thoughts, but I'm just using a couple of parallel processes. But maybe some kind of limit or due to the IPs from my research Institute... Is there any way to check out of these are genuine outages?
I guess there's no local solution then. Well then serial processing it is. Thanks
My general recommendation is to never rely in web resources. Download the required content and load from disk for analysis. Web content may change or break down, making analysis not reproducible.
Thank you for your kind answer. I would prefer that option. In fact I was not that aware that beyond the orgDB clusterProfiler relied on ensembl behind the curtains. But there is no option to user clusterProfiler offline or only with local resources, is that right?
Thanks for your time
Not that it would be directly helpful, but I remember one of the helpdesk person at Ensembl told me that server problems at Ensembl are usually reported on their blog - https://www.ensembl.info/. Additionally, depending on what information you want to extract, you could also try using the REST-API feature of Ensembl.