By Josh Cohen, PhD, Deputy Director & Chief Science Officer
This post originally appeared on LinkedIn Pulse and is reposted here with permission.
I wrote at the end of January about our research Center’s first experience publishing an article in a journal (Gates Open Research) that is both open-access (all articles are freely available to the public) and open-peer review (review occurs after initial publication, and reviews are published along with the article’s first release version). We have published a final version of the article in the journal. This update describes our experience.
The good news is that all three of our reviewers approved our initial version of the article. That means that it could be indexed in PubMed as written. Nonetheless, we worked hard to address the comments we received. Importantly, I believe that we worked with more rigor and thoughtfulness than we otherwise would have had the paper undergone conventional (unpublished, anonymous) peer review. Here’s why.
First, because the reviewer comments are published along with our article, we felt compelled to address points raised by the reviewers that we think would resonate with general readers of the journal. That is, publication of the comments put us in the position of having to “sell” our handling of those points not only to one individual (the editor at a conventional journal), but to the broader audience. The reviewers raised many good points, and we felt we had to address them to be taken seriously by our peers.
Second, it is our impression that the peer review comments were more helpful and more constructive than comments we typically receive, perhaps in part because of their public nature. We were fortunate that highly-regarded experts in our field agreed to conduct these reviews. But it seems plausible and perhaps likely that the reviewers, consciously or unconsciously, stepped up their efforts because they knew that others would see what they wrote, and that their names would be attached to their reviews.
In short, for both the reviewers and for us, there was heightened accountability to our colleagues and to others who might read this publication. And accountability is a great motivator.
One of the most important comments we received noted that the data extract accompanying our original publication appeared to contain errors. We produced that extract using data from our original database, creating it only for the purpose of this publication, as Gates Open Research requires articles to be accompanied by release of all supporting materials, including data. Fortunately, because the errors affected only the extract, they had no substantive implications for our findings. We have corrected the extract that accompanies version 2 of the publication.
The experience highlighted the importance of producing science that is not just methodologically sound, but also fully transparent and reproducible. That Gates Open Research requires open publication of all supporting materials had the effect of holding us to a higher standard than we are accustomed to. We have learned from the experience and will be incorporating these quality measures into our other work.
Acknowledgments: The Bill and Melinda Gates Foundation, which sponsors the Gates Open Research journal, also supports the research that the author and the Center for the Evaluation of Value and Risk in Health at Tufts Medical Center published in Gates Open Research.