load.rnb.set throws multiple warnings when uploading an object

Question

[Assa Yeroslaviz]

Im testing the Bisulphite-Seq workflow with RnBeads. When trying to save the rnbs object locally I use the following commad

rnbs <- save.rnb.set(object = rnbs, path = rnbsSaveDir)

When doing so it deletes the object from memory, which is a very nice function. While doing so it throws two warnings

Warning in doTryCatch(return(expr), name, parentenv, handler) :
  restarting interrupted promise evaluation
Warning in doTryCatch(return(expr), name, parentenv, handler) :
  restarting interrupted promise evaluation

Would it be better to create an uncompressed output folder?

But for me to continue the analysis, it mean I need to reload the object from memory.

When I do this I get a lot more warnings:

> rnbs <- load.rnb.set(path = file.path(data.dir,"rnbSets.zip"))
There were 50 or more warnings (use warnings() to see the first 50)
> warnings()
Warning messages:
1: In FUN(X[[i]], ...) : NOTE: did not overwrite file 'ff8fd9db6abc1.ff'
2: In FUN(X[[i]], ...) : NOTE: did not overwrite file 'ff8fd953c612ef.ff'
3: In FUN(X[[i]], ...) : NOTE: did not overwrite file 'ff8fd9710533e0.ff'
4: In FUN(X[[i]], ...) : NOTE: did not overwrite file 'ff8fd9c90f118.ff'
5: In FUN(X[[i]], ...) : NOTE: did not overwrite file 'ff8fd97fc46519.ff'
6: In FUN(X[[i]], ...) : NOTE: did not overwrite file 'ff8fd936c98dd7.ff'
...

I'm not sure what these warnings/messages suppose to tell me. Do I need to worry about something here, or are these just notifications of what is being uncompressed during the loading f the object?

Thanks Assa

>sessionInfo()
R version 3.6.3 (2020-02-29)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Mojave 10.14.6

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
 [1] grid      stats4    parallel  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] RnBeads.hg38_1.18.0                     RnBeads_2.4.0                           plyr_1.8.6                             
 [4] methylumi_2.32.0                        minfi_1.32.0                            bumphunter_1.28.0                      
 [7] locfit_1.5-9.4                          iterators_1.0.12                        foreach_1.5.0                          
[10] Biostrings_2.54.0                       XVector_0.26.0                          SummarizedExperiment_1.16.1            
[13] DelayedArray_0.12.3                     BiocParallel_1.20.1                     FDb.InfiniumMethylation.hg19_2.2.0     
[16] org.Hs.eg.db_3.10.0                     TxDb.Hsapiens.UCSC.hg19.knownGene_3.2.2 GenomicFeatures_1.38.2                 
[19] AnnotationDbi_1.48.0                    reshape2_1.4.4                          scales_1.1.0                           
[22] Biobase_2.46.0                          illuminaio_0.28.0                       matrixStats_0.56.0                     
[25] limma_3.42.2                            gridExtra_2.3                           gplots_3.0.3                           
[28] ggplot2_3.3.0                           fields_10.3                             maps_3.3.0                             
[31] spam_2.5-1                              dotCall64_1.0-0                         ff_2.2-14.2                            
[34] bit_1.1-15.2                            cluster_2.1.0                           MASS_7.3-51.5                          
[37] GenomicRanges_1.38.0                    GenomeInfoDb_1.22.1                     IRanges_2.20.2                         
[40] S4Vectors_0.24.4                        BiocGenerics_0.32.0                    

loaded via a namespace (and not attached):
 [1] colorspace_1.4-1         ellipsis_0.3.0           siggenes_1.60.0          mclust_5.4.6            
 [5] base64_2.0               rstudioapi_0.11          bit64_0.9-7              xml2_1.3.1              
 [9] codetools_0.2-16         splines_3.6.3            scrime_1.3.5             knitr_1.28              
[13] Rsamtools_2.2.3          annotate_1.64.0          dbplyr_1.4.3             HDF5Array_1.14.4        
[17] readr_1.3.1              compiler_3.6.3           httr_1.4.1               assertthat_0.2.1        
[21] Matrix_1.2-18            prettyunits_1.1.1        tools_3.6.3              gtable_0.3.0            
[25] glue_1.4.0               GenomeInfoDbData_1.2.2   dplyr_0.8.5              rappdirs_0.3.1          
[29] doRNG_1.8.2              Rcpp_1.0.4.6             vctrs_0.2.4              multtest_2.42.0         
[33] nlme_3.1-147             preprocessCore_1.48.0    gdata_2.18.0             rtracklayer_1.46.0      
[37] DelayedMatrixStats_1.8.0 xfun_0.13                stringr_1.4.0            lifecycle_0.2.0         
[41] rngtools_1.5             gtools_3.8.2             XML_3.99-0.3             beanplot_1.2            
[45] zlibbioc_1.32.0          hms_0.5.3                GEOquery_2.54.1          rhdf5_2.30.1            
[49] RColorBrewer_1.1-2       curl_4.3                 memoise_1.1.0            biomaRt_2.42.1          
[53] reshape_0.8.8            stringi_1.4.6            RSQLite_2.2.0            genefilter_1.68.0       
[57] caTools_1.18.0           rlang_0.4.5              pkgconfig_2.0.3          bitops_1.0-6            
[61] nor1mix_1.3-0            lattice_0.20-41          purrr_0.3.4              Rhdf5lib_1.8.0          
[65] GenomicAlignments_1.22.1 tidyselect_1.0.0         magrittr_1.5             R6_2.4.1                
[69] DBI_1.1.0                pillar_1.4.3             withr_2.2.0              survival_3.1-12         
[73] RCurl_1.98-1.2           tibble_3.0.1             crayon_1.3.4             KernSmooth_2.23-16      
[77] BiocFileCache_1.10.2     progress_1.2.2           data.table_1.12.8        blob_1.2.1              
[81] digest_0.6.25            xtable_1.8-4             tidyr_1.0.2              openssl_1.4.1           
[85] munsell_0.5.0            quadprog_1.5-8           askpass_1.1

[Assa Yeroslaviz]

Thanks for the reply.

The run is finally complete, after almost 30 hours. First, I must say that I really appreciate the logging parameters. They are very informative.

But (also) because of that, I have a few follow-up questions.

> rnb.run.analysis(dir.reports = report.dir, sample.sheet = sample_annotations, 
+                  data.dir = bed.dir, data.type = "bs.bed.dir", 
+                  build.index = TRUE,save.rdata = TRUE, initialize.reports = TRUE)
2020-04-29 07:46:16     5.6  STATUS STARTED RnBeads Pipeline
2020-04-29 07:46:17     5.6    INFO     Analysis Title: Arno A. Bi-Sulphite Seq data analysis
2020-04-29 07:46:17     5.6    INFO     Initialized report index and saved to index.html
2020-04-29 07:46:18     5.6  STATUS     STARTED Loading Data
...
2020-05-01 00:49:27    13.0    INFO     Initialized report index and saved to index.html
2020-05-01 00:49:28    13.0  STATUS     STARTED Saving RData
2020-05-01 01:03:59    13.0  STATUS     COMPLETED Saving RData
2020-05-01 01:03:59    13.0  STATUS COMPLETED RnBeads Pipeline

1st Q. Why do I get these warnings?

2020-04-29 10:02:18    17.0  STATUS                     STARTED Region type: sites
2020-04-29 10:03:23    15.0 WARNING                         Could not cluster rows of the heatmap with method manhattan and linkage average . Discarding row dendrogram
2020-04-29 10:03:26    15.0 WARNING                         Could not cluster rows of the heatmap with method manhattan and linkage average . Discarding row dendrogram

This is happening with sites, with tiling1kb, ensembleRegBuildBPall, tiling200bp, gencode22promoters, gencode22genes.

Why can't it create the clustering?

2nd Q. When it is done, it print out these messages

    Warning in gzfile(file, "wb") :
    cannot open compressed file 
    '/Users/yeroslaviz/ownCloud/ownCloud_Personal/ownCloud_projects/BrendaSchulman/ArnoA/BSSeq.Analysis/Analysis/.Rproj.user/shared/notebooks/B4DE3490-RnBeads/1/2C8DEDD0F127A992/cule9xx1w5xqu_t/918fe5ff791d4ea7832b2c0a3a012d7d.snapshot', probable reason 'No such file or directory'
    Error in gzfile(file, "wb") : cannot open the connection
    Error in (function (which = dev.cur())  : 
    QuartzBitmap_Output - unable to open file '/Users/yeroslaviz/ownCloud/ownCloud_Personal/ownCloud_projects/BrendaSchulman/ArnoA/BSSeq.Analysis/Analysis/.Rproj.user/shared/notebooks/B4DE3490-RnBeads/1/2C8DEDD0F127A992/cule9xx1w5xqu_t/_rs_chunk_plot_001.png'

I know you mentioned above, we should just start a fresh R session, but is it also necessary here? Did it completed the analysis? What are these error means?

It is not really helpful to have an analysis running for two days, only to have to restart it again with no meaningful changes.

I would appreciate your help.

[GenoMax]

Please use ADD COMMENT/ADD REPLY when responding to existing posts to keep threads logically organized. This comment should have gone below @mscherer's answer.

SUBMIT ANSWER is for new answers to original question.

Answer 1

[mscherer]

Dear Assa,

Sorry for the delay in replying to your question.

You are facing a quite common issue that occurs from RnBeads' behavior of storing large matrices on disk using the ff-package. After multiple loadings and savings of the same object to/from disk, the error with file permission that you report can occur. Typically, it helps just to start a new R-session and everything should run smoothly again.

Considering the warnings that you report; they always occur when loading an object and can be ignored, since they originate from the ff-package, but do not harm the object integrity. I think the easiest thing to continue for you is just to start with a fresh R-session.

Hope that helps,

Michael

Answer 2

[mscherer]

I will try to answer you questions, but I should comment that the things you report are not really common. Parts of this answer are also wild speculations.

Question 1: The warning/error message is shown, in case there is an issue with the clustering/distance metric of the clustering using the dist and hclust functions. Since you are using bisulfite sequencing data, my speculation is that it is caused by too many missing values in you matrix. You can potentially resolve this issue by providing more stringent coverage filtering criteria using the RnBeads options filtering.coverage.threshold and filtering.missing.value.quantile.

Questions 2: It seems at if some of the paths you provide is not writable, but I guess that all output is properly generated. I would need to have some further information on how the output reports look like to determine what the exact issue might be. Could you at least point out in which of the RnBeads modules the error occurs?

Hope that helps a bit.