A multilevel cross-classified modelling approach to peer review of grant proposals : The effects of assessor and researcher attributes on assessor ratings
Journal article
Jayasinghe, Upali W., Marsh, Herbert W. and Bond, Nigel. (2003). A multilevel cross-classified modelling approach to peer review of grant proposals : The effects of assessor and researcher attributes on assessor ratings. Journal of the Royal Statistical Society Series A: Statistics in Society. 166(3), pp. 279-300. https://doi.org/10.1111/1467-985X.00278
Authors | Jayasinghe, Upali W., Marsh, Herbert W. and Bond, Nigel |
---|---|
Abstract | The peer review of grant proposals is very important to academics from all disciplines. Although there is limited research on the reliability of assessments for grant proposals, previously reported single-rater reliabilities have been disappointingly low (between 0.17 and 0.37). We found that the single-rater reliability of the overall assessor rating for Australian Research Council grants was 0.21 for social science and humanities (2870 ratings, 1928 assessors and 687 proposals) and 0.19 for science (7153 ratings, 4295 assessors and 1644 proposals). We used a multilevel, cross-classification approach (level 1, assessor and proposal cross-classification; level 2, field of study), taking into account that 34% of the assessors evaluated more than one proposal. Researcher-nominated assessors (those chosen by the authors of the research proposal) gave higher ratings than panel-nominated assessors chosen by the Australian Research Council, and proposals from more prestigious universities received higher ratings. In the social sciences and humanities, the status of Australian universities had significantly more effect on Australian assessors than on overseas assessors. In science, ratings were higher when assessors rated fewer proposals and apparently had a more limited frame of reference for making such ratings and when researchers were professors rather than non-professors. Particularly, the methodology of this large scale study is applicable to other forms of peer review (publications, job interviews, awarding of prizes and election to prestigious societies) where peer review is employed as a selection process. |
Keywords | Australian Research Council; cross-classified models; grant proposal funding; interrater reliability; multilevel modelling; peer review process |
Year | 2003 |
Journal | Journal of the Royal Statistical Society Series A: Statistics in Society |
Journal citation | 166 (3), pp. 279-300 |
Publisher | Blackwell Publishers Inc |
Oxford University Press | |
ISSN | 0964-1998 |
Digital Object Identifier (DOI) | https://doi.org/10.1111/1467-985X.00278 |
Scopus EID | 2-s2.0-0141888420 |
Page range | 279-300 |
Funder | Australian Research Council (ARC) |
Research Group | Institute for Positive Psychology and Education |
Publisher's version | License All rights reserved File Access Level Controlled |
Output status | Published |
Publication dates | |
Online | 04 Sep 2003 |
Publication process dates | |
Accepted | 01 Jul 2002 |
https://acuresearchbank.acu.edu.au/item/89417/a-multilevel-cross-classified-modelling-approach-to-peer-review-of-grant-proposals-the-effects-of-assessor-and-researcher-attributes-on-assessor-ratings
Restricted files
Publisher's version
85
total views0
total downloads10
views this month0
downloads this month