This is part of a series on publishing (in academia and/or tech).
I recently chatted with Corey Quinn on an episode of his Screaming in the Cloud podcast (#325: Into the Year of Documentation). I was delighted (although not surprised) that Corey was familiar with the “Reviewer 2” phenomenon to the extent that he dropped in a reference when describing the trickiness of writing for Twitter:
You can get into a lot more trouble tweeting than you can in academia most of the time. Every Twitter person is Reviewer 2. It becomes this whole great thing of, “Well, did you consider this edge corner case nuance?” It’s, “I’ve got to say, in 280 characters, not really. Kind of ran out of space.”
He also returned to the concept when customizing his closing comments for the episode:
If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment telling me how terrible this was and that we should go listen to Reviewer 2’s podcast instead.
For those of you who are unfamiliar with Reviewer 2, they have their origins in the peer review processes that accompany certain types of publications (qualified as “peer-reviewed” or “refereed”) in research-oriented spaces like academia and medicine, and across various disciplines (including computer science). The purpose of peer review is to ensure that a piece of writing or research meets the standards of a given publication or publisher as well as any accepted quality or implementation conventions of relevant scholarly fields.
While there are different types of peer review, three of the most common forms are
- double-blind peer review: the authors do not know the identity of their reviewers and vice versa
- single-blind: authors do not know the identity of their reviewers, but the reviewers know the identity of the author(s)
- Authors and reviewers are made aware of each other’s identities; this is often called “open peer review”
Because double-blind and single-blind peer review both require that reviewers remain anonymous, reviewers are given stand-in titles, especially when there is more than one reviewer (which most refereed publications require). Common designations involve giving reviewers numbers, ergo: “Reviewer 1”, “Reviewer 2”, etc.
It is worth noting that the feedback generated in peer reviews is often shared only with prospective authors and personnel at a given publication venue. Unlike reviews intended for public consumption–such as published book reviews (like this one) or those left by users in spaces such as Goodreads or on bookseller websites–peer review work often has an audience limited to editors and the author(s) of the reviewed piece, and, as regards the latter, reviewers usually remain anonymous.
As one can imagine, this opens the door for all types of unmitigated vitriol, self-promotion, and general bad-faith commentary among what is intended to be (and in many cases actually is) expert evaluation and helpful critique.
Indeed, the real power of Reviewer 2–the quality that has made Reviewer 2 part of our research mythos–is not that sometimes reviewers do not like a submitted piece (although this happens); it is that often one reviewer in the mix will be enthusiastically supportive of a piece while another will rip it apart if not outright reject it. And for reasons that are part of the great mysteries of the universe, the latter is somehow always (just ask any researcher) Reviewer 2.
Reviewer 2 has been known to do any or all of the following (or so I have been told):
- Recommend that the author read what the author suspects is Reviewer 2’s own research on the topic and revise the entire piece accordingly (with a tone that expresses sufficient awe for Reviewer 2’s work)
- Recommend that the author read the author’s own research and revise accordingly (this is one of the more entertaining potential outcomes of the double-blind peer review process)
- Ask for absolutely contradictory revisions (the actual implementation of which could potentially create a rhetorical paradox that results in the universe collapsing in on itself)
- Provide feedback that suggests that Reviewer 2 did not, in fact, read the piece very carefully
- Torment an author with revision requests even when the reviewer asked the author to submit the piece in the first place while knowing perfectly well that the author was super busy and had a million other things to do
https://twitter.com/ianfjanssen/status/1502989123149303812
The entire concept of Reviewer 2 seems to have taken on a life of its own. Reviewer 2 has a Twitter account (and hashtag, of course). They have inspired posts such as this one on How Not to Be Reviewer 2 (which in turn alerted me to the fact that there is an entire Facebook group called “Reviewer 2 Must be Stopped”). They have even inspired publishers to post advice pieces on What to Do about Reviewer 2.
And they have given us so many good memes, which I will leave you to uncover (or create) on your own, but I am partial to this one, as it captures one of my fears:
Criticism aside, there is much to recommend the peer review process. It is the bedrock of a reproducible scientific methodology. Also, for specialist scholarship it provides academics with much needed (and wanted) feedback from the author’s peers. In fact, I sometimes suspect that reviewers are the only ones who took the time to read my research. In the age of vaccine hesitancy, peer review is the antithesis of the dangerous “do your own research” crowd.
I would like to close with a very important point: monetary compensation for peer review work often ranges from nothing to very little. (I have heard tales that sometimes you get discounts or purchase credits towards materials produced by the publisher?) Rather, for academics it is considered part of the “professional service” component of their workloads (which I describe here), and is often seen as something one does to help keep the wheels of the profession turning.
Furthermore, it takes considerable time, care, and skill to review what is often dense and detailed information (to the point where folks have published on what it takes to be a good referee), all through the lens of someone who has spent significant time building up expertise in their chosen field. And there are many reviewers (such as the majority of those who have provided feedback on my own publications) who go above and beyond in generating clear, helpful comments designed to help authors improve their work; these reviewers are just never Reviewer 2.
nota bene: My direct experience with academic publications has been primarily with those formats and conventions related to the humanities and technical communication in the U.S. Also, I have engaged with such publications as a consumer, author, and reviewer, but not as a publisher or editor.
No Comments