Nearly 30% reported that they used particular software because it had been "validated against other methods in peer-review publications."
The article touches on the need for peer review of source code and distinction between equations / algorithms published in the paper and the implementation in code. It's an interesting read, and I'm pretty sure I agree with everything, although I think it will be practically difficult to find referees capable of evaluating both the scientific content as well as the correctness of the software. I'm preparing a small software paper, and I would love to submit somewhere like Journal of Open Research Software, but the readership seems lacking.
Software development has a particular curse: the difficulty and complexity of developing any piece of software is commonly and ubiquitously underestimated. Even software developers tend to do that let alone scientists outside of computational fields. This leads to the situation where it is more difficult to get funded on a grant that emphasizes software quality versus one that emphasizes novel discovery.
A second pitfall is that reviewing a software for correctness is a far more challenging task than reviewing a typical paper. The work that needs to be done is more taxing and is not something that is easy to switch in-and out of. Validating results may be as difficult as running a full analysis. Considering that majority of reviewing is done for free we can see how having to verify a software for correctness is not all that attractive task.
Another challenge is that a surprising number of publications describing tools do not even make their source code available. I have had several experiences recently, where I was reviewing a tool publication, requested the source code and the authors belligerently refused to provide it. They were able to cite journal policies that only require binary code to be made available. I agree, appreciation of the importance of good software engineering in science has many curses...
Another challenge is that a surprising number of publications describing tools do not even make their source code available. I have had several experiences recently, where I was reviewing a tool publication, requested the source code and the authors belligerently refused to provide it. They were able to cite journal policies that only require binary code to be made available. I agree, appreciation of the importance of good software engineering in science has many curses...