In my experiences, algorithm development is mostly about choosing a balance between features, accuracy, speed and memory. For every gain in one of these aspects, you lose in others. It is nearly impossible for a program to beat others in all metrics. All you should do is to choose the balance point you feel comfortable. For example, for ChIP-seq and CNV discovery, you do not need great accuracy. Bowtie would be the best. For the discovery of structural variations, you have to use a slow mapper to be very accurate.
However, there are exceptions. Very occasionally, smart scientists may achieve breakthroughs, when a new algorithm can excel in all aspects: features, accuracy, speed and memory. There are few breakthroughs in the area of sequence alignment I am familiar with. All I can think of are: sse2-sw, hmmer3 and bwt-sw.
As to metagenomics, my feeling is it has not attracted enough top programmers (I saw your previous question, but I agree with others that this is not true). Unless you have a really talented scientist in the field, competitions between multiple groups are necessary. The boost of short-read mappers and the recent improvement to javascript engines are among the best examples.
In addition, you should not always look for programs that can run on a desktop computer. As I said above, you often pay something else, mostly accuracy, as the trade-off. Most successful programs choose a balance point that fit a moderate workstation or server.
EDIT: If I had to work for a metagenomics project, I would have two choices: a) writing a quick-and-dirty pipeline integrating existing software developed for other purposes; b) spending most of my time on a new package tuned for the project. If I felt metagenomics was the future and everyone would go for it in a year, I would choose b); if I felt this was only a small field, I would choose a) because b) is not worth the time. I believe many developers/programmers are thinking the same way.
As far as I know, TCS and especially research in algorithms is as healthy as ever. Whether people in the industry care about their results, or whether researchers themselves are enough inclined to bridge the gap between theory and practice, is another question.
Sure Eric. Got your message. Will be careful next time. Any ways do you know of any nice fora where I can ask/discuss and get perspectives on such topics. Thanks in advance
From my talking with others, people who generate results for industry do not care too much about results, but those (e.g. service providers) who generate results for academy do care. In addition, it is worth noting that most successful/popular programs have bridged gap between theory and practice.