In my role as associate editor of a specialty journal, in the recent week or so I’ve been given, over and over, the crappiest of gifts: that two-line nonreview review, accompanied by a recommendation to reject. Some of these came from people who really should know better. These reviews are worthless. I cannot use them, because they contain no information on which I could base my decision. I cannot reject and forward these non-reviews, because they are callous toward the authors who put a lot of work into their manuscripts, and because, if I were to use them and forward them, I would show myself to be callous, too.
The question is: How does one learn to write useful referee reports/reviews of the technical work of others? (Other than being an editor, I also received recognition as an outstanding referee from a professional society, so I’d like to think I know a little bit about the topic.)
My students get training on how to write referee reports. After they’ve published a paper or two, they will get solicited for reviews whether they know what they’re doing or not, so it’s my job to make sure they do a service (rather than a disservice) to the scientific enterprise. First, I share (through the group’s password-protected document repository) a few of my own reviews with group members, which they can use as a style and structure guide. We also do joint reviews — certain society journals have this option, in particular for a supervised co-review with a (named!) junior person, whereupon the junior person also gets added to the reviewer base. I might ask a student to supply an additional review on papers where I am associate editor and the topic is in their wheelhouse. I have never, ever asked a student to do an uncredited review instead of me: either we co-review or I decline and recommend them as a reviewer, but they always get properly credited for their work.
Before I proceed, a couple of disclaimers. In my field, there is only single-blind review (the referee is not known to the authors, but the authors are known to the referee). In most journals I referee for, the referee is explicitly asked to recommend a course of action (e.g., reject, resubmit with minor/major revisions, accept as is, transfer to another journal, and sometimes other finer steps in between); these are advisory to the editor, who can ultimately do what they like. If your field or subfield is not like that, for example if you’re actually forbidden from making even a hint of recommendation, please do not assume that all fields are like yours or that I don’t know what I am talking about.
One part of the training is helping students understand what it is that they should be recommending based on the report. Are the edits minor, but mandatory? Are they minor, but optional? Then say so. Can you envision the authors making certain edits that would eventually make you happy to recommend publication? Then tell them clearly what they need to do, even if it’s major, and don’t recommend rejection, but major revisions. Is there something in the paper that’s a complete deal breaker, so that you cannot envision how it would become appropriate for the journal without being a completely different paper? That’s a rejection. For example, the paper is not novel (all they claim to have done new had already been done by others, whom they didn’t cite); the paper is wrong or fraudulent or grossly misleading; the paper is poorly aligned with what the journal publishes (e.g., outside scope; too specialized for a generalist journal).
This is how I recommend to structure a review:
a) Summarize the paper in one to two sentences. If the paper were a screenplay, this would be the logline. You’re letting the authors and the editor know what you think the paper is about.
b) A few sentences regarding your general impressions of the paper. Not just the bad; the good, too. For instance, if it’s was written well or if it’s an engaging read, say so. If it’s an interesting topic, say so. If it is timely, say so. If it is a paper on a topic that was beaten to death 10 or 50 years ago, say so.
c) A clear recommendation and the general reason for the recommendation, which should flow out of point b).
Btw, points a-c are usually one, maybe two paragraphs total. Not very long if you know what you are doing.
d) The specific points (with detailed arguments and references!) that led to your recommendation. I usually number these, and many referees do, so they’re easier to refer to during revision. I think of this list as a contract between the referee and the authors. If the authors do a good job of addressing these specific requests, you should be willing to recommend publication. I am royally pissed with referees who move these goalposts. We had one like that a few years ago. We’d address all the many, many minutiae, only to have the referee come back asking for more and new stuff. It was maddening.
Doing d) properly takes time, but this is the heart of the review, and needs to be done properly for both the authors’ and the referee’s sake. In recent months, I have had the misfortune of reviewing several manuscripts where the authors pretended whole subfields didn’t exist and they didn’t cite anyone, presumably out of ignorance, but there’s also a nonzero chance it was on purpose, in order to elevate the perceived novelty of their work. In one notable example, it took me weeks to write the review, first because I was too pissed to write it, second because I had to sit down and look up key references to show them what they were missing. I explicitly had to say it was not an exhaustive list, that it was their job to do a literature review, but I pointed them toward where they could find out more.
The style of the review should be polite and matter-of-fact; being blunt is fine, but taking jabs, especially ad hominem, is not. I occasionally catch myself being snarky and have to drop the review until I can write more dispassionately. Don’t be mean or snarky. There are always junior scientists who poured their heart and soul into the paper. Do not be cruel; it can crush a young person’s spirit and contribute to them leaving science. The editors who let mean-spirited reviews through (presumably from untouchable giants in the field) are also to blame.
Miscellaneous: Do not be a pronoun jerk. When reviewing single-author papers, it’s easy to use “the author” and never even use a pronoun. It’s inclusive, avoids misgendering people, and prevents you from appearing blatantly sexist. (Based on my single-author-paper days and even today, when I submit solo-PI proposals, there’s always someone who relishes a bit too much in using the female pronouns to tell a female author that she sucks or that her work is garbage.)
Also, when you respond to referee reports, use “Referee A/1” and “they” because you have no idea who reviewed. While I’m used to being referred to as a he, presumably because my reviews display such kickassery and competence that no one can fathom them having sprouted from a feeble lady brain, I am definitely pleased when the male gender is not assumed.
Academic blogosphere, what are your thoughts on becoming a good referee and on the peer-review process in general?
xykademiqz 
In graduate school, I think I wrote one or two initial drafts for my advisor but never saw the final drafts / saw follow-up. I now try to have trainees collaborate on one with me showing them the final draft and explicitly mentioning the reasons I made changes, like I do with writing. As is probably the sign of good training, it’s often more work than just doing it myself.
The hardest line, and one I think I’m only getting a good feel for now as an independent faculty, is the difference between “Massive revisions” and “Rejection outright,” especially in standard-level society journals where “not that important” isn’t a reason to reject. I like the idea of a referee report as a contract – do X, it will be accepted. But it’s difficult when X can be a whole class of things – especially “writing explaining points A, B, C (and others, this is not a complete list) is incoherent.” I can easily see authors thinking that the goalposts move when, from my perspective, it’s that the first draft wasn’t clear enough for me to recognize the real scientific problems.
I like that you share your reviews as templates for your group. Also, how do the supervised co-reviews work out? Are the junior reviewers named (acknowledged) in the final publication, or perhaps in an annual reviewer list published by the journal, or only named to the editorial board?
How funny— our Wednesday post is on this exact same topic.
After seeing the garbage reviews submitted by my peers I realized I was putting way more effort into writing the review than anyone else and have trimmed my effort accordingly. I just felt like I wasn’t getting much out of all that time and effort. My reviews are still 2-3 times as detailed as most other reviewers. I do realize that taking this seriously makes the peer review process work, but… I don’t have a day to devote to this every time I’m asked.
This is good advice on writen peer reviews, but I’m pretty sure it’s a gross violation of expectations of confidentiality to just post all yr reviews on a lab server where anyone in yr lab can read them.
Rheophile: I think you answered your question (how to tell a difference between major revisions and rejection). For major revisions, you, as the referee, have to be able to see a clear path toward a satisfactory revision. If you cannot see one, for example if the paper is so confusing that you can’t really tell what’s wrong with it yet, that’s a rejection. Authors should not be “editing by peer review.” It’s not the referee’s job to polish someone’s crappy early draft.
Positive Definite: No formal acknowledgement per se; basically, if they (students) do a review, the editors know they did a review.
nicoleandmaggie: Great minds think alike! Looking fwd to your post.
omdg: I’ve become relentless at rejecting requests. It’s so liberating! It’s a default swift no, unless there is a very good reason to say yes. I still review plenty, but it’s much more enjoyable, and I can devote the needed time to those I do accept.
Michael Nitabach: I have maybe half a dozen old reviews,1-2 examples for each recommendation (reject, minor/major revisions, acceptance) and without specifying which papers these were for or sharing those papers, posted for the purposes of education on a password-protected site accessible only by me and my group. I am not violating anyone’s privacy and the reviews themselves are my intellectual property. But yeah, sure, let’s go with me being the worst, although, after 10 years of blogging, I’m pretty tired of your shenanigans in my comment section.
For any new folks reading, and I cannot stress this enough:
A swift no with recommendations for someone else to review is ALMOST (like 98%) as good as you saying yes. Don’t hold on to the request for a week and then say no, if you’re going to say no, say no right away. And recommendations for other people to review, especially people who might benefit from having the report on their cv (upper level grad students, assistant professors who haven’t been discovered as reviewers, people who haven’t reviewed for the journal before), will make the editor super happy.
I’ve never shared my reviews with my students for privacy concerns. What I’ve done is to have students review an existing paper that they are reading as part of their literature review. There are plenty of badly written papers out there 🙂 to practice on.
Something I don’t actually know how to handle well in reviews is the extent to which I should be correcting the authors’ grammar. Is this something that the editor is supposed to take care of? Sometimes, when the paper is written by a non-native speaker of English, there will be obvious and glaring errors that are easy to spot. Am I supposed to point these all out?
@TCL– if it is a general problem and not just one or two instances, it’s sufficient to say that the paper needs a proofreader, possibly a professional one. One of the last papers I handled we had them do that for the conclusion section and they sent a certificate saying it had been done, even though we didn’t really need that.
If it’s one or two instances, in economics we point those out in the “Minutia” section (that I will talk about on Wednesday…) If it’s a general problem, it belongs in the “Minor” section. (Unless they make the paper so unreadable it’s a reason for rejection, then it goes in “Major”).
I do it more or less the same way as you do – and I think that most people who really like writing will put in this effort, because how can you NOT help but edit and provide corrections and suggestions.
I once had a person in my lab who was trained by an older (male) PI and this person had a very different opinion on how to do a review. They thought it wasn’t their idea to provide suggestions for improvement (like the missing literature etc that you pointed out as an example) or propose specific ways of addressing a major point. And of course I was wrong because their former PI taught them the right (old white male) way.
By the way, I recently reviewed a paper that had used a scientific editing service. First time I saw that. It was a breezy read – that’s for sure. Made it a lot harder to find the flaws because I am a sucker for a good story. I just don’t hope that this is where science will go and if it does I am switching careers.
Be specific! Include page/line numbers where specific changes should be made. Nothing is worst than the vague “the overall clarity could be improved.” The same should be reciprocated by the authors when they reply to reviewer comments “The following sentence “[insert quote]” was added to Page X, Line Y.”
A challenge I find is when I recommend reject (and thus my comments are intended to just improve the paper when they submit elsewhere), but the editor gives major revisions. In that instance, my original comments weren’t particularly exhaustive (my mentors have said that if you recommend reject, don’t spend a *ton* of time on the review) and thus my second review may illuminate more issues (esp if the authors didn’t do a great job of responding to the reviews). Also sometimes the manuscript itself is problematic, but the topic is really important and so the first review may be to just get it more readable and the second review is to make it more publishable.
I had a manuscript I was asked to review five times. The first time I recommended major revisions, and like by the 3rd I recommended rejection bc the authors were blatantly ignoring my feedback or refusing to make the changes. I kept trying to contact the editor bc I was so frustrated and didn’t know what to do. The editor NEVER responded to me. When they sent me the 5th request to review I said no and said I didn’t feel like I could be useful to the process any longer given the above.
I recently got the response letter from an editor for a different paper I’ve been asked to re-review, and I noticed that the editor mentioned in their letter to the authors that both reviewers were highly respected experts in the area. I’ve never seen that in a letter before and I appreciated it. I felt like some of my issues in the paper in the paragraph above was that the authors were rejecting my expertise, and yet I am very much an expert in the area in which they were writing. Though as I am a postdoc, I doubt myself 3000%. So this vote of confidence from the editor was really nice.
Re your last point, at least once I’ve seen a paper I reviewed include in the published acknowledgements a thank-you to the referee for “his” helpful comments. Eyeroll and sigh.
My least favorite review type is where the reviewers ask me to add in extra words (for instance, unnecessary usage of “that”) or mess with the reference style. Next least favorite is the 2-page review with lots of tiny word changes. It’s a nightmare with major revisions to keep track of the line number changes.
Actually, my least favorite is the review with no decent feedback, but these others are a close second.
Great post! These are valuable points to attend to & to teach. I’d like to second nicoleandmaggie’s suggestion that if you can’t review, give some recommendations & respond right away.
Thanks, very helpful! I’ve reviewed a fair amount (am 12 years past PhD) but still trying to improve. I may tweak my introductory section to reviews to more closely match yours. I am definitely on the end of the spectrum that puts in a fair amount of time and gives a lot of feedback, especially if the lead author is junior.
Interesting about point C: “A clear recommendation and the general reason for the recommendation, which should flow out of point b).” The journals in my field tend to have C presented as a multiple choice question, and based on a similar writeup by someone in a similar field to mine, I’ve left out point C because that other writeup indicated the prose/text portion of the review was best left without a recommendation. I’ll have to think about how to handle that in the future. And maybe spend 10-15 minutes trolling back through some of the other reviewer reports on manuscripts that I’ve reviewed (several of the journals that I review for make all reviewer feedback visible to other reviewers – at least if uploaded as text rather than an attachment).
As I’ve hit mid-career, and counted all the reviews I’ve done, I’m starting to feel less guilty about not saying “Yes” to reviews. Feeling less guilty has helped me more quickly decline so that the editor can move on.