In my role as associate editor of a specialty journal, in the recent week or so I’ve been given, over and over, the crappiest of gifts: that two-line nonreview review, accompanied by a recommendation to reject. Some of these came from people who really should know better. These reviews are worthless. I cannot use them, because  they contain no information on which I could base my decision. I cannot reject and forward these non-reviews, because they are callous toward the authors who put a lot of work into  their manuscripts, and because, if I were to use them and forward them, I would show myself to be callous, too.

The question is: How does one learn to write useful referee reports/reviews of the technical work of others? (Other than being an editor, I also received recognition as an outstanding referee from a professional society, so I’d like to think I know a little bit about the topic.)

My students get training on how to write referee reports. After they’ve published a paper or two, they will get solicited for reviews whether they know what they’re doing or not, so it’s my job to make sure they do a service (rather than a disservice) to the scientific enterprise. First, I share (through the group’s password-protected document repository) a few of my own reviews with group members, which they can use as a style and structure guide. We also do joint reviews — certain society journals have this option, in particular for a supervised co-review with a (named!) junior person, whereupon the junior person also gets added to the reviewer base. I might ask a student to supply an additional review on papers where I am associate editor and the topic is in their wheelhouse.  I have never, ever asked a student to do an uncredited review instead of me: either we co-review or I decline and recommend them as a reviewer, but they always get properly credited for their work.

Before I proceed, a couple of disclaimers. In my field, there is only single-blind review (the referee is not known to the authors, but the authors are known to the referee). In most journals I referee for, the referee is explicitly asked to recommend a course of action (e.g., reject, resubmit with minor/major revisions, accept as is, transfer to another journal, and sometimes other finer steps in between); these are advisory to the editor, who can ultimately do what they like. If your field or subfield is not like that, for example if you’re actually forbidden from making even a hint of recommendation, please do not assume that all fields are like yours or that I don’t know what I am talking about.

One part of the training is helping students understand what it is that they should be recommending based on the report. Are the edits minor, but mandatory? Are they minor, but optional? Then say so. Can you envision the authors making certain edits that would eventually make you happy to recommend publication? Then tell them clearly what they need to do, even if it’s major, and don’t recommend rejection, but major revisions. Is there something in the paper that’s a complete deal breaker, so that you cannot envision how it would become appropriate for the journal without being a completely different paper? That’s a rejection. For example, the paper is not novel (all they claim to have done new had already been done by others, whom they didn’t cite); the paper is wrong or fraudulent or grossly misleading; the paper is poorly aligned with what the journal publishes (e.g., outside scope; too specialized for a generalist journal).

This is how I recommend to structure a review:

a) Summarize the paper in one to two sentences. If the paper were a screenplay, this would be the logline. You’re letting the authors and the editor know what you think the paper is about.

b) A few sentences regarding your general impressions of the paper. Not just the bad; the good, too. For instance, if it’s was written well or if it’s an engaging read, say so. If it’s an interesting topic, say so. If it is timely, say so. If it is a paper on a topic that was beaten to death 10 or 50 years ago, say so.

c) A clear recommendation and the general reason for the recommendation, which should flow out of point b).

Btw, points a-c are usually one, maybe two paragraphs total. Not very long if you know what you are doing.

d) The specific points (with detailed arguments and references!) that led to your recommendation. I usually number these, and many referees do, so they’re easier to refer to during revision. I think of this list as a contract between the referee and the authors. If the authors do a good job of addressing these specific requests, you should be willing to recommend publication. I am royally pissed with referees who move these  goalposts. We had one like that a few years ago. We’d address all the many, many minutiae, only to have the referee come back asking for more and new stuff. It was maddening.

Doing d) properly takes time, but this is the heart of the review, and needs to be done properly for both the authors’ and the referee’s sake. In recent months, I have had the misfortune of reviewing several manuscripts where the authors pretended whole subfields didn’t exist and they didn’t cite anyone, presumably out of ignorance, but there’s also a nonzero chance it was on purpose, in order to elevate the perceived novelty of their work. In one notable example, it took me weeks to write the review, first because I was too pissed to write it, second because I had to sit down and look up key references to show them what they were missing. I explicitly had to say it was not an exhaustive list, that it was their job to do a literature review, but I pointed them toward where they could find out more.

The style of the review should be polite and matter-of-fact; being blunt is fine, but taking jabs, especially ad hominem, is not. I occasionally catch myself being snarky and have to drop the review until I can write more dispassionately. Don’t be mean or snarky. There are always junior scientists who poured their heart and soul into the paper. Do not be cruel; it can crush a young person’s spirit and contribute to them leaving science. The editors who let mean-spirited reviews through (presumably from untouchable giants in the field) are also to blame.

Miscellaneous: Do not be a pronoun jerk. When reviewing single-author papers, it’s easy to use “the author” and never even use a pronoun. It’s inclusive, avoids misgendering people, and prevents you from appearing blatantly sexist. (Based on my single-author-paper days and even today, when I submit solo-PI proposals, there’s always someone who relishes a bit too much in using the female pronouns to tell a female author that she sucks or that her work is garbage.)

Also, when you respond to referee reports, use “Referee A/1” and “they” because you have no idea who reviewed. While I’m used to being referred to as a he, presumably because my reviews display such kickassery and competence that no one can fathom them having sprouted from a feeble lady brain,  I am definitely pleased when the male gender is not assumed.

Academic blogosphere, what are your thoughts on becoming a good referee and on the peer-review process in general?