Table 10 Limitations of Current Grant Review Models
From: A new approach to grant review assessments: score, then rank
Review Mechanism | Presence of Implicit Bias | Presence of Strategic Voting (based on connection to PI) | Tie Breaking Mechanism | Influence of Outlier Ratings | Sources of Reviewer Input into Recommended Funding Order | Recommended Funding Order Directly or Indirectly Determined | Level of Uncertainty in Recommended Funding Order |
|---|---|---|---|---|---|---|---|
Unblinded Panel Review Rating of Proposal Sets | Direct | Direct | None | Medium | Rating Alone | Indirectly (Summary stats of Ratings) | None |
Blinded Panel Review Rating of Proposal Sets | Indirect | Indirect | None | Medium | Rating Alone | Indirectly (Summary stats of Ratings) | None |
Rating Review of Individual Projects | Direct | Direct but Only Impacting a Few Proposals | None | High | Rating Alone | Indirectly (Summary stats of Ratings) | None |
Partial Lottery with Initial Rating of Proposal Sets | Initially Direct and then Random Selection | Initially Direct and then Random Selection | Initially None and then Random | High (Initial Rating with no Panel) | Initial Rating and then Random Selection | Initially Indirectly (Summary stats of Ratings) and then Randomly | Initially None and then Random |
Unblinded Panel Review Rating/Ranking of Proposal Sets using Mallows-Binomial Model | Direct | Direct | Integrated Score | Low | Both Rating and Ranking | Directly | Estimated Confidence Intervals |