The purpose of peer review
Open-access publishing is sometimes thought to change or threaten peer review. Now, as Peter Suber will gladly mention, open access and peer review are basically orthogonal. Open-access journals can be peer-reviewed, and toll-access journals can be unrefereed. Institutional and disciplinary repositories typically accept both refereed and unrefereed work.
Still, an oft-heard argument goes something like this: an open-access article can be commented on by anyone, and the aggregate of those comments can serve as well as or better than the existing pre-publication peer-review workflow to point up the article’s worth (or lack of worth) to its field.
All very good—if gatekeeping for purposes of quality control is peer review’s only or primary role in the process. Certainly that is the assumption I most often see, in books like this for example.
But consider this thorough examination of one peer-reviewer’s process. “I read each paper quickly to get the overall feel for its contents, and make some preliminary notes,” says Holley. “If I feel that the article is not suitable for publication, I stop there and return it with my reasons to the editor. This has happened only about three times in all my years of peer reviewing.” (Emphasis mine.)
This doesn’t suggest to me a reviewer wielding his pen like a vorpal sword to cut down the unworthy. Nor do I believe that Holley is a cupcake reviewer. No, I think Holley is pointing out another central function of peer review.
“The peer reviewer can be the author’s best friend in creating a much stronger article than the initial submission,” says Holley. “At its best, the peer review process finds errors, methodological problems, and inconsistencies while authors can still correct them—a much better alternative than learning about them in a letter to the editor.” (Added emphasis mine.)
I’ve not had anything peer-reviewed yet; getting conference proposals approved is as close as I’ve come. David’s book was peer-reviewed, however, and the process I saw was much less gatekeeping than critical feedback on making the book better. I do believe it served that purpose admirably.
If we admit that improving papers before they see the harsh light of day is one (though not the only) function of peer review, then post-publication measures come up short, don’t they? By design, they hit a paper once it’s been enshrined in the scholarly record as final.
I still think it quite possible to come up with peer-review systems that take advantage of the breadth of reviewing talent available via the Internet to improve the quality of the scholarly record while avoiding some of the cronyism, bias, and outright error that plague the existing system. Unless we acknowledge all the functions of peer review, though, whatever systems we come up with will not serve even as well as the present one.