How to Write a loop for a big matrix?

Question

[Jaan]

Hi every one,

I have a matrix with 21492 columns and 8532 rows. i am running flexmix package to calculate Gaussian univariate distributions. When i run the following commands the process will crash due to lack of memory, but if i run the first 5000 column [X <- as.matrix(df[,1:5000])] only, I get the results.

How can I write a loop for mult.models.fit1 object such a way that, the loop goes through all 21492 columns by reading the subsection of matrix's columns (lets say 3000 columns by 3000 to the last one), with out affecting the over all result of the function "stepFlexmix" itself??!!

>set.seed(1234)
>library(flexmix)
>load(".../MACompendium.RData")
>df <- as.data.frame(eset)
>X <- as.matrix(df)
>dim(X)
[1] 21492  8532
>mult.models.fit1 <- stepFlexmix(X ~ 1, k = 1:6, model = FLXMCmvnorm(), nrep = 15, control = list(minprior = 0)
1***************
2*************error: could not allocate the vector size 1.7 G.B

[David W]

Do you have to keep whatever "eset" is at first, and the dataframe and the matrix version of it, in memory?

[Jaan]

Thanks for the interest.

No I don't need to put the eset (an expression set from Microarray data) nor df in the memory, as soon as i convert it to a matrix (X) version.

[David W]

Well, rm(eset, df) will free up some room. From those dimensions, each object will be > 1Gb in memory.

More generally, I doubt it's possible to you can't split the calculation in multiple chunks and stitch the result back together, the EM algorithm works on an entire data-set.

[Jaan]

Thanks for the heads up, yes i was having the same doubt, and i was searching for a way to over come that EM Algorithm, which looks to whole data set.

By the way, i am running this data set on our server (100 GB memory). I did not use rm(eset, df) because i taught i have enough space if i dont, but no harm to remove them.

[David W]

Your error message says you are running out of memory (http://stat.ethz.ch/R-manual/R-devel/library/base/html/Memory-limits.html) there are some packages for using "disk as memory" but I don't know if they'll work with flexMix