Heatmap In R With Large Datasets

Question

[ChIP]

Hi!

I am sure, many of you guys at some point have experienced this problem in R.

I have a data matrix of 17 * 20500 (R*C)

I was trying to make an heatmapo of this, using R

Using following commands

rpkm<-read.table("heatmap_table.txt", header=T)
 row.names(rpkm)<-rpkm$Gene
 rpkm<-rpkm[,2:16]
 rpkm_matrix<-data.matrix(rpkm)
 pdf("rpkm.pdf")
 heatmap(rpkm_matrix)

and I get an error

Error in lapply(args, is.character) : node stack overflow
Error in dev.flush() : node stack overflow
Error in dev.flush() : node stack overflow

How can I fix this? please help.

Sample Data (tab delimited)

Gene    A    B    C    D    E    F    G    H    I    J    K    L    M    N    O
PQ1    7.3159    9.3802    10.77    8.701    13.6066    8.3253    9.0556    9.8801    9.0776    11.2029    7.61    10.8403    9.2378    12.1697    9.7482
PQ2    7.4715    5.2955    10.2275    6.3606    10.1463    5.9968    6.2673    8.6119    6.153    6.7903    4.0843    13.0875    6.8167    8.3186    6.7643
PQ3    0    0    0    0    0.0026    0    0    0    0    0    0    0    0    0    0.0037
PQ4    1.776    1.125    1.3508    1.2489    2.1252    2.1057    1.0177    1.6063    1.0053    0.9571    1.4972    1.3998    1.0935    2.4737    1.2063
PQ5    0.1024    0.092    0.0473    0.071    0.1227    0.2047    0.2481    0.1089    0.0499    0.1381    0.057    0.0953    0.0433    0.0651    0.0598
PQ6    5.4296    0.1688    2.4767    0.2507    0.5087    4.2835    2.2989    8.6027    3.1126    0.4565    0.167    2.9066    3.195    0.942    5.8904
PQ7    0.2918    11.5673    4.9554    0    1.6693    1.6301    0.4985    2.4444    0.6217    1.4638    3.2648    0.5773    3.1071    7.651    0.4068
PQ8    0    0    0    0    0.0575    0.1018    0    0.0422    0    0    0    0.0257    0.0276    0    0
PQ9    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
PQ10    18.789    24.8681    29.8037    33.3986    37.8269    24.4719    21.1101    26.9985    21.9897    25.3416    26.77    23.1337    20.5193    27.0328    23.9777
PQ11    0    0.004    0    0    0.02    0    0    0.0265    0.0348    0    0.032    0    0.026    0    0.0167
PQ12    2.8442    4.7904    5.8717    3.2287    5.0917    1.5291    4.1187    6.207    3.532    5.4896    5.7066    5.1487    6.4386    11.2159    7.3469
PQ13    0    0.12    0.1776    0    0    0.0366    0.027    0    0    0    0.0462    0    0    0    0
PQ14    0    0    0    0    0    0    0.0136    0    0    0    0    0    0.0083    0    0
PQ15    0    0    0    0    0    0    0    0    0    0.0322    0    0    0    0    0
PQ16    0.0321    0.0469    0    0    0    0    0.0342    0    0    0    0    0    0    0    0
PQ17    0    0    0    0    0.0466    0.0225    0.0619    0.284    0.1252    0.0205    0    0.0371    0.1413    0.018    0.1238
PQ18    9.2029    12.3713    12.0135    7.7052    9.9121    7.3582    9.6782    12.5931    9.8137    12.4413    11.3418    14.4504    7.9965    8.5895    6.1705
PQ19    16.6408    9.1365    13.8613    12.6089    12.2094    16.5078    22.4689    16.4531    16.2172    15.6118    14.8256    18.5057    16.5483    13.5991    15.4934
PQ20    26.6048    24.1932    25.3238    27.9098    29.5022    25.4348    31.1095    30.4802    28.4243    21.2893    18.7577    27.3286    26.2074    30.6207    25.0771
PQ21    1.1114    0.067    0.3146    0.4593    0.3675    2.773    0.8119    0.5015    0.4696    0.5876    0.1406    0.2492    0.8565    0.2326    0.1521
PQ22    7.4962    5.3051    8.9577    6.1617    8.5887    8.2902    7.0983    7.1107    6.0231    6.9078    6.6685    7.3996    7.3611    8.3344    5.5536
PQ23    13.596    7.4782    9.6589    6.3121    10.7004    8.5035    9.769    10.1801    6.7358    5.0971    6.2171    9.713    7.0575    10.0523    7.5863
PQ24    18.564    35.9577    30.4134    27.9277    41.4544    23.1528    15.4656    32.0211    24.979    24.7365    41.5781    28.6164    34.8429    37.6385    27.1767
PQ25    15.5685    17.3154    17.0986    10.2068    13.5607    8.281    8.57    14.104    8.5732    8.3098    15.7368    18.3766    14.6625    14.2864    12.3646
PQ26    3.6639    5.5865    6.4437    2.7832    4.6902    6.4854    3.305    4.8913    3.0334    4.1835    5.9565    5.0441    4.4169    6.005    3.5551
PQ27    0.2116    0.0035    0.1193    0.0462    0.1113    0.3879    0.2976    0.9519    0.3039    0.0613    0.0478    0.5218    0.3197    0.1381    0.2277
PQ28    32.5026    28.1368    28.2335    25.6904    36.3761    26.779    36.4265    30.5154    35.1618    23.8327    27.087    24.7966    29.477    30.0189    26.1931
PQ29    1.8439    1.4574    1.2994    2.4006    0.6938    2.7233    0.6461    0.5976    1.7659    3.4405    1.5791    0.3336    1.8652    1.6685    2.0173
PQ30    1.7028    0.9633    2.0401    1.4563    1.4204    3.7509    1.843    2.071    2.3559    2.3659    1.2402    2.0673    2.2783    2.4221    1.3163
PQ31    0.1401    0.0283    0.3815    0.0434    0.1124    1.0891    0.0681    0.3404    0.2097    0.0552    0.1386    0.1835    0.2828    0.2267    0.2176
PQ32    3.1838    2.1398    4.1528    1.9499    3.0831    3.6193    3.0609    4.4113    2.4607    1.604    3.2404    4.4924    3.0917    4.525    3.0178
PQ33    0.0187    0.042    0.107    0    0.0162    0.0114    0.0366    0.0467    0.0532    0    0    0    0.0703    0.1173    0.0472
PQ34    1.3782    0.1604    0.3452    0.2124    0.0376    0.7386    0.4819    2.5638    0.3134    0.2188    1.6717    1.2121    0.4294    0.2202    0.2482
PQ35    0.0634    0.0294    0.0735    0.005    0.0558    0.1777    0.1734    0.0536    0.0259    0.0459    0.0217    0.0388    0.073    0.0206    0.074
PQ36    7.3565    4.5738    4.9642    1.8203    4.8537    12.1248    12.4298    8.541    11.8094    12.964    7.1189    17.0531    10.7116    6.5249    15.9312
PQ37    19.2056    16.5482    10.3252    26.8747    30.8489    26.5403    27.2519    12.1769    34.8122    26.1242    14.3651    12.6533    43.6538    24.7434    19.5469
PQ38    1.4191    5.2542    2.7084    4.6994    2.6367    3.0067    3.2322    3.8202    3.6902    3.6689    3.5244    1.3118    6.2961    3.4399    4.7755
PQ39    0.0032    0    0.0419    0    0.0001    0.0044    0    0.0259    0.0059    0    0    0    0.0203    0    0.0142
PQ40    5.5934    1.2258    7.0247    3.1928    3.7698    14.0234    2.3485    6.2129    4.1372    2.4574    3.8062    5.01    3.4968    4.6268    2.7763
PQ41    0.0664    0.0029    0.984    0    0.0448    0.3315    0.0709    0.5556    0.066    0.0443    0.1812    0.0918    0.1818    0.0491    0.2315
PQ42    12.4147    11.7431    20.2819    16.29    13.8172    16.5791    5.4218    11.46    15.264    26.2695    21.1681    14.128    16.8515    15.1775    11.4873
PQ43    0.0047    0.001    0.0731    0.0118    0.0169    0.207    0.0649    0.9764    0.0626    0.0002    0.0034    0.0657    0.3199    0.0003    0.2807
PQ44    0.135    0.0166    0.6497    0.0055    0.0229    0.1664    0.1529    0.4149    0.0361    0.0109    0.255    0.1788    0.1709    0.0291    0.3004
PQ45    56.8427    37.853    26.6238    10.5706    33.1238    45.9608    13.0512    17.1816    17.2876    12.7038    48.581    57.7831    20.1544    55.8307    17.7855
PQ46    0    0    0.0638    0    0    0    0    0    0    0    0    0    0    0    0
PQ47    0.3183    0.5558    0.9872    0.7507    0.963    0.9077    0.5323    2.3656    0.9466    0.8255    0.3479    1.184    1.8744    0.6751    0.3804
PQ48    0.0887    0.0237    0.5628    0.0256    0.3346    0.3528    0.1441    0.7293    0.2763    0.1582    0.0346    0.2104    0.3426    0.2687    0.152
PQ49    0    0    0.006    0.0455    0.0238    0.0174    0.0009    0.0376    0.0125    0    0    0.0417    0.054    0.0259    0
PQ50    6.836    4.0512    3.4106    10.1931    8.6312    11.5594    5.885    7.575    7.1599    6.9806    4.6518    11.5319    11.749    8.0642    10.8055
PQ51    21.7114    18.2957    15.3453    12.8093    16.7419    17.639    20.8327    19.5234    16.4421    13.332    18.3224    22.835    21.2342    19.8245    22.699
PQ52    2.3514    5.6845    5.4899    5.7975    5.1881    4.596    2.5575    3.9446    3.8788    7.3335    5.0485    3.6145    4.1323    3.1613    3.1188
PQ53    0.4847    0.9983    1.0557    1.0674    1.606    0.8673    0.9555    1.3153    0.8204    0.8006    1.1499    0.8388    1.0022    1.7286    0.9965
PQ54    3.915    7.084    9.5129    7.2756    8.8139    7.5316    7.7541    6.6888    6.2194    11.9289    6.6386    5.3701    7.666    6.9825    6.4324
PQ55    0.0521    0.0073    0.0856    0.0077    0.0169    0.0127    0.058    0.0101    0.049    0.0051    0.0666    0    0.0311    0.0264    0.0196
PQ56    0    0    0    0    0    0    0    0.0122    0    0    0    0    0    0.0036    0
PQ57    0.1148    0.0342    0    0.4443    0.136    0.1045    0.0843    0.1647    0.132    0.1781    0.0125    0.201    0.3315    0.3318    0.0784
PQ58    38.9165    40.1913    38.3052    26.8888    47.0209    23.5061    41.6533    27.1291    34.4119    50.4297    44.3284    37.291    42.2802    43.7717    50.6826
PQ59    0.4955    0.8122    1.3064    0.6068    0.6497    1.1965    0.7193    0.8926    0.7059    1.5431    0.6225    1.2769    0.9994    0.7831    0.8051
PQ60    0.091    0.3673    0.1676    1.2076    0.0685    0.0516    0.0952    0.3293    0.1623    0.87    0.0547    0.0124    0.4719    0.0739    0.3426
PQ61    1.1572    4.0768    1.4677    1.8966    1.6605    1.2624    0.6203    2.0909    2.5919    1.4544    1.9626    1.3565    2.9001    4.1942    1.2521
PQ62    7.3755    10.2023    10.6945    10.269    5.4028    8.4086    7.3205    11.0645    3.3731    10.3396    7.4639    10.6587    7.7363    10.4975    9.4922
PQ63    0    0    0    0    0    0.0147    0    0    0.0154    0    0    0    0    0.0137    0.0295
PQ64    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
PQ65    6.5592    2.7869    6.0986    2.0528    1.9756    3.905    6.847    3.9042    4.3978    3.5496    5.7033    4.7661    2.4534    8.1218    5.9046
PQ66    0.075    0.0731    0.1346    0.0077    0.1332    0.0445    0.0566    1.6151    4.4815    0.1189    0.0136    0.1523    1.2194    0.996    0.4762
PQ67    0.0632    0    0.2768    0    0.0367    0.3175    0.1483    1.1521    0.135    0.0023    0.002    0.1234    0.2125    0    0.369
PQ68    3.6368    6.4187    5.5612    7.8563    5.4043    8.4435    7.6765    3.5599    4.4661    9.9604    6.843    3.8869    5.9325    5.9034    5.0855
PQ69    0.0193    0.1142    0.1476    0.1855    0.201    0.2492    0.1252    0.885    0.2013    0.3745    0.0594    0.7768    0.4287    0.3814    0.0767
PQ70    6.303    6.3139    7.6156    4.8289    7.1431    6.0122    7.4721    7.1121    7.2523    5.5666    9.3483    9.4402    10.2338    7.3652    6.7345
PQ71    15.23    16.7022    15.9912    16.4834    22.0234    16.6245    15.7218    17.0682    12.1024    14.9979    12.8247    22.4266    16.0012    15.9931    15.3725
PQ72    19.279    18.8995    19.0535    9.9956    18.5529    10.3627    13.8918    15.8052    18.9067    10.0083    15.1752    20.0399    21.1724    22.2839    16.8733
PQ73    26.0938    22.123    25.7748    19.7764    35.4732    29.3767    32.9773    27.0604    37.6029    23.8149    23.029    27.2217    26.9024    25.2986    30.8695
PQ74    10.5552    18.0196    13.8072    11.4033    19.2606    14.4272    14.3193    15.6022    15.7561    12.6645    15.9548    14.6876    16.1394    9.3908    15.0279
PQ75    16.0106    15.3039    16.2289    17.4721    22.205    17.0142    15.5222    14.9413    18.4688    17.9412    16.0356    18.5253    14.1441    17.5067    15.0929
PQ76    2.1649    2.19    3.522    0.3134    0.7608    1.6    6.2902    1.8681    4.2319    1.4238    2.5591    1.9849    5.586    7.5534    3.174
PQ77    0.6345    0.0504    0.0314    0.4073    0.1469    0.316    0.1119    0.6915    0.2011    0.0679    0.0232    0.4057    0.6567    0.2577    0.3413
PQ78    0    0    0    0    0.0138    0.0207    0    0    0.0374    0    0    0    0    0.011    0.0229
PQ79    0.0555    0.0205    0    0    0.0062    0.0245    0.0293    0.0729    0.0198    0.0225    0    0.0061    0.0541    0.0351    0.0364
PQ80    0.0484    0.0071    0.011    0.0072    0.0558    0.16    0.0786    0.0188    0.0264    0.0437    0.0339    0    0.061    0.0394    0.0265
PQ81    17.2016    12.6523    15.0314    8.4666    14.0292    10.7709    7.2803    12.4842    7.3373    6.6757    12.7611    19.7932    10.7824    10.8537    10.1101
PQ82    1.0711    1.2235    1.1401    0.7059    2.8462    1.1418    0.6808    1.6061    1.958    2.8307    1.6332    1.6401    1.9621    1.9076    0.8659
PQ83    4.7948    3.9261    2.1783    2.299    2.0716    4.9755    4.004    4.1771    1.6487    5.0873    1.9055    5.1117    6.0549    2.8892    4.596

Thank you

[James]

Without some idea of the data content it is difficult to imagine the problem. Can you post some example data on pastebin or some reproducible example R code that demonstrates the problem? There is almost the same problem here: Error in heatmap.2 (gplots) which suggests that non-informative data (too many rows containing identical values) might be the source of the problem. Some thoughts meanwhile: Does it work with a smaller subset of data -- i.e. only 2000 rows; what OS are you using; are you using 32-bit R; how much memory is available to your R session?

[ChIP]

Hi James,

I have put the sample dataset in the question, I am working on a very strong linux server at my dept. I am using 64 bit R. The program runs smoothly for a small dataset say 2500.

I think the problem is of large number of rows and I need a solution to fix that.

[Devon Ryan]

The example dataset you posted, at least, works properly. Do you get the same error when using that?

[Irsan]

What are you trying to achieve? Do you want to perform unsupervised hierarchical clustering of your samples? You don't need to cluster your genes for that and you don't need a heatmap. If you really want a heatmap then make one from informative genes, those whose expression values are associated with a phenotypic trait. If you don't have phenotypes available (but I hope you do) then select e.g. 1000 genes with the highest variance across all samples.

Answer 1

[seidel]

You can fix it by reducing your data set. Unless you have a display that's at least 20,000 pixels wide you won't be able to see the result anyway (even an iMax HD theater is only what, 10k?). Heat maps are usually a combination of clustering (somehow) and visualization. Do you really need both of these applied simultaneously to your data set (i.e. by using the heatmap function)? OR do you want to isolate the data in your matrix that looks interesting and visualize just that?

If you apply a filter to your data, such as looking at variance across rows or columns, and discard those that show very little or no variance, this will reduce your set for visualization and clustering. Such a step can be an informative classification between the stuff that changes and the stuff that doesn't (and you get to be in charge of how to evaluate the line). Either way, I think you should be very clear as to why you might need to visualize the entire data set at once, versus reducing it.

[ChIP]

Agree, now how about a small example of applying variance to discard the sets that show very little or no variance.

Could please put that in the set of commands I am using, just for sake of example and guidance.

Thank you for showing the way.

[Irsan]

# Calculate the RPKM variance between your samples:
rpkm_variance <- apply(rpkm_matrix,1,var)
# plot the variance for each feature (in your case protein I think)
plot(sort(rpkm_variance))
# Decide on N based on your plot. Take the top N variable features, I will use top 250 most variable features for this example
N <- 250
top_variable_genes <- names(sort(rpkm_variance)[1:N])
rpkm_variance_top_variable_genes <- rpkm_matrix[top_variable_genes,]

Answer 2

[umer.zeeshan.ijaz]

This is precisely why I wrote HEAPcloud

Usage: http://userweb.eng.gla.ac.uk/umer.ijaz/bioinformatics/HEAPcloud.pdf

Check the demo here.

I was faced with the same problem of viewing huge datasets. If you can convert the abundance table to "comma-separated" format, try it out in HEAPcloud and use browser's zoom in-zoom out to adjust the resolution. I am not sure If it will fry the browser. Probably wont because you have on 15 features in the second dimensions (which get animated as a word cloud).

Answer 3

[Alex Reynolds]

Group or categorize your genes and split your heatmap table file into separate files by these groupings or categorizations. Make heatmaps from each subset.

Answer 4

[greg]

See my posts at the R bug tracker and Stack Overflow for a mechanism to address this problem.

-Greg