How to Write a Manuscript Review

This one was inspired by a recent conversation in my group meeting.

Generally, the outcome of a review of a manuscript in the physical sciences is one of the following options (I am sure it’s basically the same in the biological and social sciences, and maybe even in the humanities, but I have no direct experience):

a) Accept as is

b) Reconsider with minor revisions

c) Reconsider with major revisions

d) Reject

As a referee, you will be asked to submit a report (to be transmitted to the authors and thus to be written in a collegial manner) along with a recommendation to the editor on the course of action. The recommended course of action is one of the options a) to d). Some journals offer further recommendation subtypes, such as “Accept with optional revisions” vs “Reconsider with mandatory revisions (minor)” vs “Reconsider with mandatory revisions (major)” vs “Reject and recommend transfer to another journal”.  Some have additional options for the referee, such as “I don’t need to see the paper again” or “I need to see the paper again.” But these are all nuances.

Reviews (report plus recommendation) are advisory to the editor. Again, reviews are advisory to the editor. Whether the review is positive or negative, write it so that the editor can understand what has governed your recommendation.

Each referee’s recommendation after a round of review is somewhere between a) and d). Based on these recommendation, the editor makes a  single decision between a) and d) — to accept as is, to invite resubmission with minor or major revisions, or to reject. This single decision is communicated to the authors. The authors will also see the referee reports, but may or may not see the recommendations of individual referees (depending on the field culture and journal), but the authors can usually tell from the content of the reports what each referee recommended. We focus on referee reports and the associated recommendations here.

When do you recommend “Accept as is”?

Option a), “Accept as is,” is usually not recommended by any referee after the first review unless the referee doesn’t give a toss. (It’s okay to “Accept as is” after the first or second revision.) As an associate editor, when I get an adulatory but shallow one-liner after the first review, “This paper is great, publish as is,” I roll my eyes. Such a report is completely useless. It offers me no advice, other than the advice that you as a reviewer didn’t take your job very seriously. Don’t be that reviewer. If you like a paper, your one-liner will not help against a scathing three-page report of another referee who hated the manuscript. If you really like the paper, give the authors something they can use to fight for it.

When do you recommend “Reconsider/accept with minor revisions”?

When you generally like the paper and its conclusion. You think the study is correct, the figures are clear, the conclusions are supported by the data, and the paper is written well. You were able to follow what they did and how, and you have enough information to determine that the technique is appropriate and correctly applied. The minor revisions are usually: missing relevant references (a small number), minor instances of unfortunate wording, some minor tangents that would be interesting to address as they link the paper to the broader field in a way the authors didn’t consider, clarifications in the title or abstract or intro, other clarifications of specific pieces or wording or details in the technique (experimental conditions, theoretical parameters), minor corrections to the figures (e.g., recommendation to choose different colors for better contrast in a 3D plot). Basically, the paper would not be awful to be published “as is” but it could be improved to full awesomeness with edits that are not overly time-consuming.

When do you choose “Reconsider with major revisions” vs “Reject”?

Is there plagiarism/duplication of work? If yes, reject, and provide references where the overlapping work has appeared.

Is the paper topically inappropriate for the journal? If yes, then reject, and explain briefly why it doesn’t fit (these are often caught by the editors, so the paper is desk-rejected).

Is the paper not hot enough for the highfalutin journal? If the answer is affirmative, then reject, but please please explain why you think so. A negative one-line review is just as useless as a positive one. The editor can’t do much with your “gut feeling” that the paper is not cool enough for the journal, especially if that’s your only reason to reject the paper. (Unfortunately, what the gut of famous Prof. Greybeard has to say seems to have more weight than in the case of younger folks). Your gut feeling should in principle be translatable into human speech, such as: all the references are old and there are no new ones, so this work is not timely enough for this journal; most of the references, especially recent ones and/or the ones with similar work, have been published in this other journal instead; the results are straightforward extension of published work and thus of very limited novelty; the results require unrealistic parameters or only occur under a very narrow set of conditions and are thus likely not robust, etc. [see comments for differences among fields]

Are the methods without a doubt inappropriate to address the problem at hand? Then reject. But if the method is one of several and is just not what you would use, that’s not a good enough reason alone to reject the work. Different methods have different strengths and often reveal different facets of the same phenomenon.

Now we come to the tough region.

Is the paper correct? Are the methods appropriate? Is it timely? Is it interesting? Does it present something novel about the world that is not obvious?

If the answer to all these questions is yes, then ask yourself if you can envision this paper being edited so as to become publishable. What would the authors have to do, specifically, to make it suitable for publication?

Does the language need considerable attention? Is the discussion of the techniques/methods unclear? Are the conclusions unclear? Can you write down what specifically is unclear?

If the answer is that you just hate all of the paper, that it’s boring or just awfully written, or that there are comprehensive, pervasive changes in every aspect, then please reject outright and try to explain ghat the paper is far from publishable form and that you cannot imagine it becoming publishable within the span of 1-2 revisions, that it would essentially have to become a completely different paper instead because of simultaneous issues with presentation, conclusions, figures, etc. It is much better to reject outright than to 1) torture yourself to try to list all the things that are wrong, 2) make the authors spend a lot of time entering those edits, only to 3) find out that even after all these edits you still think the paper is awful. Rejecting a paper because of pervasive issues is a kindness. For instance, imagine if you were to submit a first draft of paper written by a second-year graduate student. These drafts usually require extensive edits and the advisor has to make several (many?) layers of corrections in order for it to become suitable to unleash upon the world. Similarly, there is no point in wasting the time of multiple referees “editing by peer review” something that’s as far from publishable as an early draft of a newbie student.

So when do you say, “Reconsider with major revisions”? When you can envision a finite number of specific things that the paper needs in order to become publishable. Imagine receiving the paper with those revisions perfectly incorporated; if you would then have no problem accepting the paper, then that’s major revisions. Major revisions usually include: significant gaps in cited literature, missing data/figure(s) in order to support a conclusion, missing critical information that prevents a reader from following the exposition or assessing the correctness of the approach, poorly written abstract or conclusion.

How do you write a useful referee report? 

Start with a 2–3-sentence-long paragraph (Hyphen happy! Technically, the first one is a dash.) in which you state, in your own words, what the paper is about, how the authors do what they do, and what the main findings are. This helps show the authors and the editor that you have understood the paper.

Then say clearly, in a single-sentence paragraph,  what your position on the faith of the paper is. Do you feel it’s generally great, but have minor suggestions for improvement or minor but required edits? Do you think it’s inappropriate for publication in the present form, but expect it to become publishable if the authors satisfactorily address the specific problems outlined below? Or do you think the paper is simply not appropriate for publication in This Journal for reasons that are deal-breakers, and concisely explained?

If your are disposed towards rejecting, make sure you state why in a few sentences or a couple of numbered items/paragraphs.

If there are major issues with the paper, give a numbered list of major issues that the authors should address. Be specific about what you want them to do. Remember, if you are a good referee, this should be like a contract: if they do what you ask, you will recommend acceptance. Don’t be that douche who keeps moving the target and asking for new and varied things in subsequent reviews. Follow up by a list of minor concerns, like the typos you caught, unfortunate wording, missing units, etc.

If you have identified minor or optional revisions, list them also in a numbered list. If something is optional to consider and you do not require that the authors comply, but just to seriously consider it, then say so.

Happy reviewing!

(See the comments for some differences between fields, and of course your own advisor is the best guide for what the practices are in the field your work in.)

13 comments

  1. Great post! I’ll refer my students to this as a reference from now on.

    I have two other suggestions for new reviewers. First (and this should go without saying) be respectful and polite, even if you are criticizing the manuscript. I have received some truly nasty comments in reviews. This can be extremely demoralizing to new researchers, and is inappropriate. Also stick to comments that are 1) constructive and 2) about the work and not about the authors. Your review may be blind to the authors, but the editors know who you are!

    My second suggestion is that if as a reviewer you think that a result is widely known, please suggest at least one reference to back up this assertion. If the result is really widely known, this should be easy. If instead it is something “everybody knows” but no one can point to any data in support, it might be worth publishing.

  2. Great post! One suggestion for minor revision 😉
    For most journals I review for much, there is a separate text box in the form for “comments to the editor”; this is where the clear 1-sentence recommendation to the editor goes. The rest of the review you paste into a separate text box as “comments for the authors”. Many journals request you not to put the explicit recommendation for acceptance, revisions, or rejection into the comments to authors, since they are advisory to the editor, who may agree with you or not on the final decision, and the journal may wish to present a more unified front to the authors in the event of dissenting opinions. In any event, the body of your review in “comments for the authors” should provide sufficient explanation to both the editor and authors for why your recommendation was what it was.

  3. We recently sent a brief paper to a quick publication short format at a mid-level society journal. The results were clear cut and the conclusion was interesting, and we were hoping for a quick publication before a grant review.

    Unfortunately, we got 1 reviewer of our paper that was a total loon. She insisted that we needed to do another major experiment for our study–an approach that no one except her ever does. It also was not even necessary for our conclusion, and would have provided evidence that at best was just correlatory. It was easy to identify her as the reviewer by just googling that particular type of approach since only she had ever done it in the literature–in one minor paper (as I said, the technique does not provide relevant information). Did I mention that the technique was also nearly impossible to do with our system? And even if we did the experiment, the resulting data would not fit into the short format of the paper we were proposing?

    We tried to explain in our response to the reviewer why that approach was not needed for our conclusions, and was almost impossible to do, and politely asked why she thought it was necessary? But she dug in her heels, refused to answer our questions about why this approach was so important, and just kept insisting over and over again to the bitter end that had to we do our study “her” way or it was not valid–and in broken English no less.

    We went back and forth 3 times with the journal, always with her responding in the same bizarre way. I met with my lab group each time, where we looked at her lengthy and heated responses written in strange syntax and said to each other “are we missing something? What in the world is she meaning here? This doesn’t even make sense!”

    In the end I got annoyed and withdrew the paper and sent it to a higher impact journal instead.

    A funny note– this reviewer actually came up to my student who was presenting this work in a platform talk at a national meeting last week—she insisted again that her approach was absolutely necessary but she again could not respond to any of my students’ questions about why. And yes, it was exactly the person who we thought it was–a minor and inexperienced scientist. Apparently she took it as a personal affront that we did our study without utilizing her favorite technique but she lacked the critical thinking skills to understand why her approach was not necessary or important.

  4. I would add that if the English is truly abysmal, you should just reject. I have reviewed (or been the editor for) several journals that get a lot of so-called international submissions (I’m not talking about top tier journals). If the language in the original ms truly sucks, the next submission will be at best mediocre, and it will take at least a third revision to get the language up to barely good enough. It’s tempting to think that if only the authors can clean up their writing, somehow their ms will be transformed into a beautiful butterfly. But it never happens. Most caterpillars stay caterpillars.

  5. Also, another way to make a more helpful review is to list exactly where in the paper you would like to see the change – Page X, Column Y, Line Z. And in reverse, the authors can refer to locations in the updated manuscript the same way. That way there is no digging/confusion and it is very clear where changes are needed/have been made.

  6. @anon: What we (my group) routinely do as authors is to denote changes via colored font in the revised manuscript (I use LaTeX but it’s trivial in Word, too), so it’s clear where something has been changed. Also, in the response letter, we do something like this: Referee’s point No 3 (verbatim), followed by authors’ response to point NO 3, followed by the changes to the manuscript in response to that point (as you say, specifying where in the text, e.g., paragraph 3 on page 7; that text will be in color — we like blue or green, but many people do it in red). Then move on to the next point by the referee.

    crystaldoc: Many journals in my field have a separate box to select the recommendation to the editor (reject, reconsider with major revisions, etc.) which the editor only sees. I still think it’s important to communicate to the authors clearly in your report what you expect them to do to satisfy you and where you as a reviewer stand. I usually say, in that second paragraph, things like “This manuscript is interesting and well written and the conclusions are supported by data. However, the authors need to address these relatively minor points prior to publication,” then list points. It’s clearly minor revisions, no? Or “This paper is not appropriate for publication in the present form because of the issues detailed below, but I expect it will become if the authors successfully address them.” Clearly major revisions, no? Or “This paper is not appropriate for publication in This Journal because [not timely, not novel, not appropriate for venue]…” Clearly a rejection.

    You don’t have to say “reject” or “reconsider with major revisions” for it to be clear that that’s what you mean. (I will now incorporate “minor revisions” in the post! 🙂 )

  7. Prodigal Academic: Completely agreed. Some people are really nasty in review; do they ever stop to think how they or their students feel when they get a similarly worded report? It can be absolutely crushing for students and, let’s face it, even we grownup scientists get bummed out by an unprofessional or mean review. In contrast, a kind and positive review can lift a student’s spirits up and re-motivate them for their work for a very long time!

    And such a great point about “it’s widely known…” We are now battling over one such paper, where we conclusively show what many people believe to be right but there hasn’t been proof yet. One of the referees has been driving us nuts vacillating between “this is known, so not worth publishing” (uh, if so, so where are the citations?) and “I can’t believe you actually showed it, give me even more details!” We eventually asked the editor to have someone else look at the paper because we felt we were being strung around. We’ll see how it ends!

    artnscience: In cases of a nutcase referee, I have in the past always asked that the editor send to someone else because the referee was not acting professionally. The editors always comply. Much of the time, getting a new referee does end positively for us. I am sorry you had to withdraw the paper; what a time sink.

  8. @Xyk: I’m sorry, but if it is the *explicit* policy of the journal that I do not communicate a recommendation as to accept, revise, reject, etc., to the authors but only to the editors, then I’m gonna comply. Of course, it will be clear from my detailed comments to the authors what my basic position is.

  9. @TOS: Do what you gotta do.

    No journal I am familiar with has some sort of ban on communicating what your recommendation is to the authors. It might be different field cultures. I cannot imagine, in any journal in my field, a referee getting in trouble for saying “I will recommend this paper for publication provided these edits are made” or something to that effect. It’s pretty common verbiage in my field.

    As a referee, I recommend what I recommend and I stand by it; the editor can do whatever they choose to do with it; they can edit out any part of my report they don’t like. As an editor, I have made a decision plenty of times when there are dissenting opinions.

  10. These days I mostly review for the JBC. From their specific instructions to reviewing editors:

    “It’s important that the “Comments for the authors” do not include explicit recommendations for acceptance/rejection/revision. The authors might receive critiques from two reviewing editors whose recommendations might not agree. While reviewing editors usually come to the same conclusions, associate editors sometimes do have to reconcile diverse views. It can be difficult for an author to accept a rejection when an explicit recommendation for acceptance after revision, for example, is embedded in the other critique. Your explicit recommendations are very welcome, but they should be made directly to the associate editor in the “Private comments to AE” section of the review form.”

    Certainly I will make it clear if there are changes that would be required to satisfy me (especially concerning the strength of the data in support of the conclusions) or specify if other recommendations I consider to be more optional suggestions. On the other hand, I am particularly careful about wording when it comes to my opinion of whether the findings are of high enough significance for the journal– there my opinion is definitely more advisory to the associate editor handling the ms. If I think significance is strong then I will highlight that in my initial summary. If I think significance is marginal or low, then I will mention that weaknesses include the incremental nature of the advance, or the significance likely limited to a narrow sub field, or something like that. I will never state in comments to authors that the work is not significant enough *for this journal*, although I may imply this.

  11. @crystaldoc: I believe you. This seems to be a difference between our fields.
    I am not sure how widespread it is.

    In my field, one is commonly asked “Is it important/hot enough for this journal?” See for instance what Physical Review Letters says. It explicitly asks to say in the report if the paper is appropriate for the more prestigious Letters than for one of the regular Physical Review journals: “PRL aims to publish innovative work of significant impact and interest. Your report and recommendation should address the basic question about any possible Letter: Why should this paper be published in PRL, rather than in the Physical Review, which also publishes papers that significantly advance physics?” More along the same lines here.

    I touched up the post to alert the reader about some of the differences between fields that we have discussed here.

  12. Great post, especially for students/postdocs.

    Regarding suggestions based on timeliness/novelty/whatever: I always take these seriously when reviewing for PRL, which I consider to be my bread and butter society journal whose prestige I want to protect.

    But when it comes to the glamor stuff like all the NPG journals I find I just don’t care. I usually don’t say much for the timliness/etc. part (but obviously still the correctness and clarity parts) and let the editors decide. I still get annoyed when crap is prominently displayed there, but I don’t feel any ownership for them.

    I also second prodigalacademic advice: it is so important to realize this m/s represents a huge portion of a person’s (often relatively young) life. Your words have a big impact, please use them to help not hurt.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s