Memory Error

Question

[cthangav]

Hello, I am getting this "unable to allocate memory" error in R when trying to do this tutorial. What is the best way to avoid this memory issue? Should I increase memory limit()?

> mofa <- run_mofa(mofa)
Connecting to the mofapy2 python package using reticulate (use_basilisk = FALSE)... 
    Please make sure to manually specify the right python binary when loading R with reticulate::use_python(..., force=TRUE) or the right conda environment with reticulate::use_condaenv(..., force=TRUE)
    If you prefer to let us automatically install a conda environment with 'mofapy2' installed using the 'basilisk' package, please use the argument 'use_basilisk = TRUE'

    Error in py_call_impl(callable, dots$args, dots$keywords) : 
  MemoryError: Unable to allocate 2.43 GiB for an array with shape (11909, 27367) and data type float64

Detailed traceback: 
  File "C:\Users\cathe\AppData\Local\basilisk\120584~1.1\MOFA2-~1.1\mofa_env\lib\site-packages\mofapy2\run\entry_point.py", line 738, in build
    tmp = buildBiofam(self.data, self.data_opts, self.model_opts, self.dimensionalities, self.train_opts['seed'],  self.train_opts['weight_views'])
  File "C:\Users\cathe\AppData\Local\basilisk\120584~1.1\MOFA2-~1.1\mofa_env\lib\site-packages\mofapy2\build_model\build_model.py", line 48, in __init__
    self.build_nodes()
  File "C:\Users\cathe\AppData\Local\basilisk\120584~1.1\MOFA2-~1.1\mofa_env\lib\site-packages\mofapy2\build_model\build_model.py", line 57, in build_nodes
    self.build_Z()
  File "C:\Users\cathe\AppData\Local\basilisk\120584~1.1\MOFA2-~1.1\mofa_env\lib\site-packages\mofapy2\build_model\build_model.py", line 91, in build_Z
    self.init_model.initZ(qmean="pca", Y=self.data, impute=True, weight_
In addition: Warning message:
In run_mofa(mofa) :
  No output filename provided. Using /tmp/mofa_20210430-004104.hdf5 to store the trained model.

[Dunois]

Are you potentially running a 32-bit version of Python? Take a look at this StackOverflow post here.

Might also just be that you didn't actually have 2.43 Gibibytes (= GiB) available to allocate to that array. Were you trying to run this on a machine with 8GB of RAM and a bunch of other stuff running in the background?

[cthangav]

It may be the latter because I am using the 64bit version. Here is the Python Information:

> py_config()
python:         C:/Users/cathe/AppData/Local/basilisk/1.2.1/MOFA2-1.0.1/mofa_env/python.exe
libpython:      C:/Users/cathe/AppData/Local/basilisk/1.2.1/MOFA2-1.0.1/mofa_env/python37.dll
pythonhome:     C:/Users/cathe/AppData/Local/basilisk/1.2.1/MOFA2-1.0.1/mofa_env
version:        3.7.7 (default, May  6 2020, 11:45:54) [MSC v.1916 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/Users/cathe/AppData/Local/basilisk/1.2.1/MOFA2-1.0.1/mofa_env/Lib/site-packages/numpy
numpy_version:  1.19.2

My computer has 8GB of RAM, 7.88 of which is usable. I have set the memory.limit() to 50000 megabytes, but not higher. I'm also working with large data objects from the tutorial that are taking up space:

It may also be because of overcommit handling mode, but I do not know if changing that is something I should do.

[Dunois]

Do you actually need that Seurat object sitting in memory? I'd try deleting that, and also running gc() and giving this a try again.

You had 7.88 GB free, but looking here, you're actually out of RAM; just that Seurat object and that MOFA object alone have eaten up ~7.2 GB (I'm assuming this is vanilla R with no hidden memory shenanigans).

I think you're going to have to manage memory a bit aggressively since you don't have much to begin with. So not a bad idea to write intermediate files out to disk if possible and removing them from RAM, and reloading the data back once the MOFA is done. I'd actually just remove everything that's not necessary for that particular function to run, and see if it works.

You could also try increasing the page file size like that StackOverflow thread recommended.

Also, best to do all of this with a fresh R session, I guess.

[cthangav]

Thank you, I was able to do it when removing the seurat object. Is this the best way to read and write objects to the disk space?

#save seurat object to disk
saveRDS(seurat, file = "seurat object")
rm(seurat)
#restore seurat object
seurat <- readRDS(file = "seurat object")

[Dunois]

I've not worked much with the Seurat package, so I can't really comment on this one. I suppose it's alright if you're able to write to file and read it back into memory fast enough?