I have to stop reviewing other people’s stuff.
Raining criticism on people really bums me out. I am in the middle of a panel (yes, again) and just feeling desperate. (It doesn’t help that I am a double token: token woman and token theorist on an experimental panel). Of the batch of the proposals I reviewed, most were really very good: well written, engaging, feasible. But, during discussion, you find that one person’s feasible is another person’s incremental. One person’s exciting and transformative and worth taking a chance on is another person’s unconvincing and needing a third mountain of preliminary data. Ripping apart good proposals so we could justify funding only 15% is a freakin’ slaughter of science. NSF funds good science; it also doesn’t fund probably 3x as much science that’s just as good.
One panelist didn’t give anyone more than a G (E-excellent; V- very good; G – good; F – fair; P – poor) and gave several people a P. G already completely eliminates a proposal from the running. Nobody should get a P, that’s just downright mean; P basically coveys that the reviewer thought you were a total moron. There were several proposals that started with scores like E/V, V, G, P. Seriously, the same proposal is both excellent and poor?
The worst thing about NSF peer review in particular is that the reviewers raise all these issues that may or may not hold water, and you have no opportunity to defend yourself. Even if the reviewer is very wrong, unless someone who understands that is on the panel, was assigned your proposal, and is also not a know-it-all a$$hole, the unfounded criticism stands and sinks a good proposal.
I want to read papers and enjoy them and admire people’s cool ideas. I am tired of hunting for things not said, or tangents not addressed; I am tired of hearing of the minor flaws others hunted down and blew up into fatal weaknesses. I want to believe that most people are serious scientists, and if they have ideas for which they painstakingly collected preliminary data and wrote a proposal on, that they are usually serious, willing, and able to to do the work.
It also makes me desperate in the face of writing more proposals of my own. Somehow, whatever I write, however I write it, it’s just not quite there. People say my proposals are good and that I just have to keep resubmitting, that the successful people wait it out, but what’s the point? If you don’t get funded the first 2-3 times, the field moves on. Formerly competitive proposals get stale.
I’m gonna get some beer at the hotel restaurant, bring it to my room, and work on panel summaries.
xykademiqz 
Amen, sister. And somehow I doubt it’s going to get any better over the next four years.
Last panel I was on used Z-scores so that different panelists could be compared. That way, someone who grades everything as P doesn’t get to sink individual proposals. Also: every granting scheme should at least include a rebuttal option to counter reviewer arguments, methinks. Also2: amen to the fact that finding excuses to be able to rank a few rare proposals as ‘best’ on a pile of excellent science is very very demotivating and sad.
This is completely different from my most recent experience on a panel, but I am not in such a male-dominated field. In fact, I would say that my field’s program officers do a fantastic job in assembling balanced panels. It makes sense that the clusters (or whatever) would develop their own cultures, for good or ill, that then affect the panels.
I disagree that nobody should get a P. Good lord, the dreck I’ve seen. The worst proposals are from people who are completely out of the loop of mainstream science, or who know better but are not making a serious effort. I assume the latter submit whatever they have on hand just so that they can tell their dean that they are trying to get funding. They deserve a P for wasting everyone’s time.
I watched a friend of mine who had this completely brilliant idea have her proposal shot down twice. I couldn’t believe they’d do it once let alone twice. First time because it didn’t have enough preliminary data and was too out there and second time because it was too incremental. I think that was the last straw for her: despite being well funded, she went to work in industry where she’s now heading an R&D department for a huge company and making four times the salary. She seems so much happier and feels appreciated, so she definitely made the right move.
haha I didn’t notice the terrible pun in your title until the 2nd time I looked! Nice.
In my experience on NSF panels, a “P” is a communication to the proposal writer that we expect better from them. On the panel I was most recently on, several proposals got P’s when they were written as “I’m famous. Give me money.” I saw P’s on an NSF panel reviewing institutes when some people stapled together their unrelated NIH proposals. Similarly, in the old NIH 1-5 system, a “5” was a communication to the PI. For example, I once saw an NRSA (a postdoc grant to work with someone) get 5’s when the famous PI signing it clearly had not even read the postdoc’s research plan.
One of the problems with the new NIH 1-9 system is that reviewers want to save “9” for that kind of message, but NIH wants us to use “9” for anything in the bottom 50 percent. Because reviewers push 9 to the bottom, it stretches all the other scores and you lose resolution in the areas it’s needed (because reviewers like linearity, but the scores are not supposed to be linear).
I can’t remember the last time I saw a throwaway proposal submitted to the NSF. Since it went to one submission per year all around, people are exceedingly careful about what they submit. Even the low-ranked proposals were really carefully put together and clearly involved a lot of work. There was one that I reviewed that showed that senior author didn’t really know how to write good proposals for this division, but even that one had objectives, background and motivation, preliminary data, tasks, and timeline, and for the most part some eye candy on almost every page. Honestly, it’s probably been a decade since I last saw something that I would really characterize as poor submitted to the unsolicited window at the NSF; maybe different communities differ.
Do you have any advice for a first-time reviewer?