Quantiles Normalization On Massive Dataset

Question

[Paul]

Hi,

I've got a massive dataset of 160 Affy Exon arrays that I need to quantiles normalize at probe level (the reasons for which are complicated!), which means ~6 million probes per sample.

Trying to load the into memory maxes out 32gigs of memory on one of our cluster nodes.

I clearly need to find a way to do this without loading it into memory, would anyone perhaps have a pointer/suggestion?

Thanks!

Answer 1

[Laurent]

A few suggestions:

• Aroma affy or xps for on-disk access instead of loading everything into memory,
• fRMA (frozen RMA) to pre-process single or batches of arrays against precomputed reference probe effects.

Hope this helps.

Answer 2

[Malachi Griffith]

I also recommend Aroma affy, but you might also consider RMAExpress