Forum:On the utility of publishing a tool paper

Question

[Matt Shirley]

I've been considering writing an application note for the pyfaidx module (for reading/writing indexed FASTA files), but I'm not sure if the effort involved in authoring and publishing an application note is worth it. Several projects have published their work as application notes, but I'm not sure that a "me too" attitude helps here.

Reasons I can think of for publishing a tool:

1. Citations. Obviously it's easier for people to reference your work.
2. Content discovery. Not everyone knows what they're searching for, and while GitHub and Google do help here, not everyone is an SEO genius.
3. Context for usage. Several application notes I've seen provide use cases or examples where the tool may provide an advantage.

Downsides:

1. Time
2. Publication fees
3. Danger of producing a stale description of your software. Software development should be motivated by use cases, bugs, and user feedback. All can really change the functionality and interface of software.

Any thoughts about pros/cons of tool publications would help.

[Chris Miller]

in academia, publications are currency. Until granting agencies and hiring/promotion committees become more accepting of alternative metrics, you will be at a disadvantage if you don't publish. Even if you toss off a quick one-page paper to bioarxiv or PLoS One, it's worth your time.

[Matt Shirley]

In favor of publishing tool papers? Leave a comment (please don't vote) here and tell me to do it!

[Mary]

I have tried to track software, sometimes very unsuccessfully, and I find it irritating to not have a point of reference. When personal web sites disappear, and university site redesigns screw up all the links, it's really frustrating to trace the details. Could you at least consider something like FigShare? It could be fine as an informal note, with a bit more explanation than just documented code. But that would not be daunting or costly. Edit to note: you can also update FigShare if there are subsequent changes or issues.

[Matt Shirley]

Hate tool papers? Leave a comment (please don't vote) here and let me know why.

[Devon Ryan]

Honestly, it depends on your job context. If publications can benefit you, then publish. Otherwise, put something up on bioarxiv (or at least make good documentation) and call it done.

[Matt Shirley]

I like the idea of providing good documentation, and I think you've correctly identified that I'm worried about job context. However, I'm not sure that it's always a good idea to publish, which is the point of my OP. I appreciate your feedback.

[Devon Ryan]

I think the time and cost are pretty valid arguments against publishing if you don't actually need publications. However the static nature of a publication is a pretty minor criticism, in my mind at least. I think that most people are pretty understanding in that regard (and if not, they know so little that who cares about their opinion).

[Jean-Karim Heriche]

As others have said, I too can see only one reason to publish a paper: to add a line to your CV. However, as you point out, a paper can help advertize your work but for this you don't need peer-review. For the same money, just buy ad space in a journal widely read by your target audience.

[Ryan Dale]

From the consumer standpoint, I see bioinformatics journals as sort of a universal venue for describing software that spans many cliques (programming language cliques; github users vs non-github users, etc). I also see publications as a coarse quality filter for code that the authors/reviewers feel is ready for general use, and therefore worth my time to check out.

From the producer standpoint, I agree with your pros. For the cons, time spent writing up an application note make you think about the big picture, puts your code in perspective, and can help strengthen your docs. And as long as you provide a link to the github repo of your code, stagnation isn't so much of an issue as long as the app note is a relatively high-level intro and description.

[Devon Ryan]

If only reviewers did a better job at actually reviewing the software being described. the quality of this seems to be really hit or miss (granted, this is true of peer-review in general). I recently tried to use a tool published 2 months ago where the tool can't perform some of the analyses in the paper that describes it and that said paper explicitly says should be possible.

[Giovanni M Dall'Olio]

If you are worried about publication fees, Application Notes in bioinformatics are not expensive. If you choose not to use the Oxford Open option, and do not print any color figure, I am not sure you have to pay anything at all.

[pld]

A pub is a pub, go for it!

Answer 1

[Istvan Albert]

For me it comes down to "personal responsibility" towards the code - say if I have a github repo with a some of software I may or may not support it long term, I may or may not answer issues or fix bugs.

I consider a software paper publication to be a declaration that I will support and maintain the software for a considerable amount of time (3-5 years at least)

[Mary]

I understand the idea of public responsibility to support the code. But do you feel any responsibility to at least get it out there because taxpayers [presumably, if you are grant funded] have the right to know what work transpired on their money? I know that is how some people feel about the availability of stuff from grant-funded research on other fronts. Seems to me even if you can't support it, there's value in the concepts, what worked, and sometimes what doesn't work, that it hard to find later if there's no record at all.

[Matt Shirley]

I think this is a vote in support of open development of software on sites such as GitHub, Bitbucket, and Sourceforge. That's something I don't dispute the utility of at all. The line of thought behind publishing a paper is that the tool might be widely useful and the authors want to make such a statement, and should probably commit to keeping the software usable and current.

[Mary]

But I see code repositories as the basic minimum. I guess I'm saying that a more narrative structure--like more typical in a publication format (formal or informal), is different to me. That provides context for the problem, reasons to have chosen one direction over another, comparisons with other tools in the space, etc. On some great github pages I've seen that--so I see it can be done there, but I've seen much more sterile styles of drop-and-run too.

[Istvan Albert]

Oh yes, everything is public and it is usable http://viramp.com/ with sources at https://github.com/ywan/viramp-project and docs at http://docs.viramp.com/en/latest/

It is just that it does not have the quality/reliability that I expect from a software that is published. We will probably work on it again once we can ensure that we can fix issues and problems in a timely manner. This paper made me realize the futility of providing Amazon AMI images - instead we should have used automated installation via ansible.

Also here is a prior post on it Virus Sequencing de novo vs reference based assembly

[pld]

I think this is a very good point and something to consider, especially in the current "political" state of funding. While it may have been useful for internal analysis, not publishing it more or less means the software (and the resources invested in it) "goes to waste".

I think this is especially true when the software allows for some new form of processing or analysis. You can think of your software being less a tool and more a proof of concept that your approach works. Having the software is great, but at the very least having some means for another person to at least obtain the concepts and explanations is great. For some very large tools or forms of analysis that require complex or non-trivial implementations I think support is a must, but for smaller tools or forms of analysis that aren't as complex to implement, the important thing is to get the idea out there it at least provides a starting point.

[Matt Shirley]

This makes a lot of sense to me. Actually, the stewardship aspect of publication makes me think the software should be designed with maintainability and API stability in mind. Then, if the tool is non-trivial to recreate by others, publication is a great option. I've decided to start a manuscript for the tool mentioned in my OP: https://www.authorea.com/users/3882/articles/14770/_show_article. Comments are welcome.

Answer 2

[robert.davey]

F1000Research has a great attitude to software papers, and supports publishing of software updates in the form of incremental DOIs.

http://f1000research.com/for-authors/article-guidelines/software-tool-articles

They offer open peer review, so your article goes straight up for public consumption and then goes through review and anyone is able to comment on your article. This is a really nice system for publishing software, in my opinion, and their turnaround times for review are pretty fast.

You may also like to check out PeerJ, FrontiersIn and PLOSOne as examples of other reputable open access publishers.

[Matt Shirley]

Thanks for the suggestions. I'm considering PeerJ and Bioinformatics. PeerJ (and F1000R) seem to be great platforms for publishing an open-access tool paper, and Bioinformatics has the readership demographic I would want to target, but with a less appealing platform and ethos.

[robert.davey]

No problem. I'm of the firm belief that the journal "impact factor" has almost nothing to do with whether a researcher will actually read and value the paper. For example, I get most of my reading material suggestions from electronic means, such as Twitter and PubMed/Google Scholar crawlers (e.g. pubcrawler) for particular topics that interest me. I rarely grab a copy of Bioinformatics short of scanning the back cover for something that catches my eye, or to find a specific paper cited in another.

I'd much rather publish in a journal that treats me well as an author, has a full (green) open access policy, and truly believes in pushing the publishing movement towards a better place from the outset, unlike traditional journals that have had to reluctantly move with the times ;)

The F1000 platform is really nice (especially when you couple it with collaborative writing services like Overleaf, i.e. WriteLatex, that seamlessly let you publish to the F1000 service), and I've yet to publish in PeerJ but it's been an intention for a while now.