What is your opinion on the network score of IPA (Ingenuity Pathway Analysis) analysis?

Question

[itsunoda]

I have a question about "network score" of IPA Network analysis. In many papers, the top 5 networks were listed in tables, while in these tables some network scores are high (around 50), but others are low (less than 20). We use the same method for network analyses, and got the impression that we can see tight association between genes when "the network score" is higher than 40. However, we have not found literature discussing the meaningful "network score" (we found one paper described that "the networks are selected if their score is higher than 21"). We would appreciate it if you could let us know information about such a meaningful network score or your impression/experience of the network score (for example, did you see tight association of genes when the network score was less than 20?).

[russhh]

Push some randomly sampled gene lists into the calculation and decide for yourself. If the same genes keep coming up in the returned networks, you might want to question its validity

[itsunoda]

Thank you for your comment. We have an impression that network scores higher than 35 seem to be meaningful. Please let me know your opinion on the network score.

[russhh]

My opinion is that it's too easy to convince yourself what a meaningful score is, in an ultimately unpublishable way. Please do some sanity checks of the form I just described. Merry Christmas

Answer 1

[emekdemir]

Unfortunately IPA has been completely opaque about its algorithms and their multiple hypothesis testing seems to be iffy at best... use with extreme caution..

Answer 2

[Charles Warden]

Last time I used IPA, the network score was just the number of genes in the network (including those in the your query gene list as well as those added through indirect interactions). There was no statistical test, and the scores will obviously vary with the number of genes in the gene lists (larger gene lists should produce larger networks).

Nevertheless, biologists often liked this function because some patterns come out when you visually inspect the networks (like maybe 80% of the genes in the network are connected to NFkB). In my opinion, I think the upstream regulator function provides a more rigourous way to identify regulatory hubs, but there point is that there are some people who like the network function (although I think it is important to realize that there is no statistical methodology behind that score)

[itsunoda]

Thank you for your comment. Accordingly, we counted the number of genes in the network (including those in the our query gene list as well as those added through indirect interactions), as you did in your last IPA usage. We found that the network score was not the number of genes in the network (it was close, though).

According to page 6 of the white paper, "IPA Network Generation Algorithm"

Step 6. The final step is the calculation of p-scores used to rank networks on the IPA "Results" page. The p-scores are derived from p-values. Say there are n gene sin the network and f of them are Focus Genes. The p-value is the probability of finding f or more Focus Genes in a set of n genes randomly selected from the Global Molecular Network. It is calculated using Fisher's exact test as described in section 3, step 6. Since interesting p-values are typically quite low (e.g.: 10-8) it is visually easier to concentrate on the exponent. Therefore, the p-score is defined as:

p-score = -log10(p-value)

I am really not sure whether "p-score" is the same as the network score. When I checked "Results" page, there is no information about "p-value."

I would appreciate your help.

[Charles Warden]

Maybe it has changed since I used the program...I don't work for IPA, but you can ask somebody from their tech support:

[e-mail]

http://www.ingenuity.com/customer-support