Comments on: NIH Challenge Grants: Is Competition Bad?

By: DocEm

DocEm — Thu, 25 Jun 2009 21:23:54 +0000

One institution put in close to 3,000 applications – with two hundred topics, I don’t see how they came up with 3,000 original ideas. Similar (“essentially equivalent”) applications are not allowed by NIH. Furthermore, NIH had to recruit applicants to review grants, including competing grants. If your employer submitted 500 grants, as a reviewer its in your interest to trash any grant because it likely competes with a colleague.

Nothing about this system seemed to work. As for stimulus, these academic institutions charge 50-70% in overhead. That would be 100-140Million towards utility, taxes, insurance, endowment, whatever, but not towards research and job creation.

Off my soapbox. Generally, I have a lot of respect for NIH, but I think the quality and review would have been higher if they enforced their own policies.

By: danielhorowitz

danielhorowitz — Fri, 22 May 2009 17:24:31 +0000

@ThaboM

I think you are mostly right, but Rafe is right too.

I do agree that the grants are likely to be awarded to those who know how to “game the system,” and write good proposals. But, the more proposals submitted, the more likely one of them is going to be a “game-changer.”

Regardless of the number of proposals, the innovative game changing ideas are unlikely to be chosen. This is the core problem that needs to be fixed.

Do you think gambling could save science?

By: ThaboM

ThaboM — Fri, 22 May 2009 08:18:49 +0000

Competition does not necessarily deliver better quality results.
As per your last post Rafe, it rewards those who have greater skills at writing proposals.
If you are skimming, you are bound to miss the value in a proposal unless it is written in a way that attracts your attention.
If you are reading carefully through the document, you are more likely to understand the concept.
Also, a truly innovative proposal will probably not survive skimming as it will have less cognitive hooks in the reviewers mind to link to.
So the end result, is more likely to be the winners will be well-written, safe proposals and unlikely to be game-changing in any way.

By: rafefurst

rafefurst — Tue, 19 May 2009 16:05:20 +0000

It’s the market in action. If you have a kick-ass proposal you will be happy to spend some cycles to try to get the money. If you have a mediocre one, then maybe you should rethink your profession.

The risk as I see it is skewing the incentive to great grant writers who are bad at doing the research they propose. But let’s jump off that bridge as we get to it. There be dragons everywhere.

By: danielhorowitz

danielhorowitz — Tue, 19 May 2009 15:53:14 +0000

@Rafe

Maybe you as the grant giving institution wants more applications but what about the other side? (scientists) Having to apply for a thousand grants to get get one seems like a waste of researchers time? (i.e. increased transaction costs to obtain money)

By: rafefurst

rafefurst — Tue, 19 May 2009 15:42:23 +0000

It’s the standard statistical argument, nothing more nothing less. More applicants means longer tails from which to choose.

As for increasing “transaction cost”, my contention is that if I have 10 applications to review I will read them all thoroughly. If I have 1000, then I will skim their abstracts and discard 99% of them based and only delve deeper into the ones that are clearly outstanding. I will ALWAYS want more applications not less. Even if I can’t possibly even read the titles of all of them (let’s say there are a million), by having larger numbers the chances that the “black swan” is in the field increases and I at least have a better chance of noticing it than if it didn’t apply at all.

By: danielhorowitz

danielhorowitz — Mon, 18 May 2009 15:10:09 +0000

@Rafe

So the more applicants the better? 200 applicants per grant is better than 100 applicants per grant? This seems like an awful lot of unnecessary transaction costs to me.

How do you measure quality? Years of experience? Education? And, why should I believe that these higher quality applicants produce better science?

By: rafefurst

rafefurst — Sat, 16 May 2009 21:20:09 +0000

Looking at the challenge areas it seems to me that it has a good chance to spur some creative and much needed research. Given that I would think that more competition means better quality applications will get funded. I don’t think there’s any inherent benefit to funding a higher percentage of applicants actually get funded. For everyone who gets discouraged and doesn’t apply, ten more will gladly apply.

@David, I think you mean “paradox of choice” right? I think it’s a valid concern for the reviewer, but I don’t think it negatively affects the quality of the choices. On average the grant quality should go up.

By: David Basanta

David Basanta — Sat, 16 May 2009 14:03:58 +0000

I wonder if this will have similar effects to the paradigm of choice. Reviewer’s (if that were to be true) would be better served having a few quality options than being overwhelmed by too many options that cannot be properly examined given time and other constraints. Just thinking aloud.