The Emergent Fool

NIH Challenge Grants: Is Competition Bad?

Posted by danielhorowitz in Asymmetry, Competition, Economics, Government, Health on May 15th, 2009 | View Comments

As part of President Barack Obama’s stimulus package, NIH is awarding $200 million in “Challenge Grants.” But, according to Science Magazine, these grants are far more competitive than initially intended:

A frantic grant-writing effort that has consumed biomedical research scientists this spring came to an end last week, resulting in a huge pile of new applications—more than 10 times larger than expected—to be reviewed by the National Institutes of Health (NIH). After this enthusiastic response, there will be many disappointed applicants: The rejection rate could run as high as 97%.

Increased time cost spent not just applying for money but also reviewing applications to allocate the funds. Additionally, many qualified researchers chose not to apply for funding after hearing how competitive the grants were. Is this a good thing? Does this competition result in better science? Could there be a better way to allocate scientific funding?

Related posts:

David Basanta

I wonder if this will have similar effects to the paradigm of choice. Reviewer’s (if that were to be true) would be better served having a few quality options than being overwhelmed by too many options that cannot be properly examined given time and other constraints. Just thinking aloud.
http://rafefurst.wordpress.com/ rafefurst

Looking at the challenge areas it seems to me that it has a good chance to spur some creative and much needed research. Given that I would think that more competition means better quality applications will get funded. I don’t think there’s any inherent benefit to funding a higher percentage of applicants actually get funded. For everyone who gets discouraged and doesn’t apply, ten more will gladly apply.

@David, I think you mean “paradox of choice” right? I think it’s a valid concern for the reviewer, but I don’t think it negatively affects the quality of the choices. On average the grant quality should go up.
danielhorowitz

@Rafe

So the more applicants the better? 200 applicants per grant is better than 100 applicants per grant? This seems like an awful lot of unnecessary transaction costs to me.

How do you measure quality? Years of experience? Education? And, why should I believe that these higher quality applicants produce better science?
http://rafefurst.wordpress.com/ rafefurst

It’s the standard statistical argument, nothing more nothing less. More applicants means longer tails from which to choose.

As for increasing “transaction cost”, my contention is that if I have 10 applications to review I will read them all thoroughly. If I have 1000, then I will skim their abstracts and discard 99% of them based and only delve deeper into the ones that are clearly outstanding. I will ALWAYS want more applications not less. Even if I can’t possibly even read the titles of all of them (let’s say there are a million), by having larger numbers the chances that the “black swan” is in the field increases and I at least have a better chance of noticing it than if it didn’t apply at all.
danielhorowitz

@Rafe

Maybe you as the grant giving institution wants more applications but what about the other side? (scientists) Having to apply for a thousand grants to get get one seems like a waste of researchers time? (i.e. increased transaction costs to obtain money)
http://rafefurst.wordpress.com/ rafefurst

It’s the market in action. If you have a kick-ass proposal you will be happy to spend some cycles to try to get the money. If you have a mediocre one, then maybe you should rethink your profession.

The risk as I see it is skewing the incentive to great grant writers who are bad at doing the research they propose. But let’s jump off that bridge as we get to it. There be dragons everywhere.
ThaboM

Competition does not necessarily deliver better quality results.
As per your last post Rafe, it rewards those who have greater skills at writing proposals.
If you are skimming, you are bound to miss the value in a proposal unless it is written in a way that attracts your attention.
If you are reading carefully through the document, you are more likely to understand the concept.
Also, a truly innovative proposal will probably not survive skimming as it will have less cognitive hooks in the reviewers mind to link to.
So the end result, is more likely to be the winners will be well-written, safe proposals and unlikely to be game-changing in any way.
danielhorowitz

@ThaboM

I think you are mostly right, but Rafe is right too.

I do agree that the grants are likely to be awarded to those who know how to “game the system,” and write good proposals. But, the more proposals submitted, the more likely one of them is going to be a “game-changer.”

Regardless of the number of proposals, the innovative game changing ideas are unlikely to be chosen. This is the core problem that needs to be fixed.

Do you think gambling could save science?
DocEm

One institution put in close to 3,000 applications - with two hundred topics, I don’t see how they came up with 3,000 original ideas. Similar (“essentially equivalent”) applications are not allowed by NIH. Furthermore, NIH had to recruit applicants to review grants, including competing grants. If your employer submitted 500 grants, as a reviewer its in your interest to trash any grant because it likely competes with a colleague.

Nothing about this system seemed to work. As for stimulus, these academic institutions charge 50-70% in overhead. That would be 100-140Million towards utility, taxes, insurance, endowment, whatever, but not towards research and job creation.

Off my soapbox. Generally, I have a lot of respect for NIH, but I think the quality and review would have been higher if they enforced their own policies.