Uncovering predictors of disagreement: ensuring the quality of expert ratings
Journal article
Hoth, Jessica, Schwarz, Björn, Kaiser, Gabriele, Busse, Andreas, König, Johannes and Blömeke, Sigrid. (2016). Uncovering predictors of disagreement: ensuring the quality of expert ratings. ZDM Mathematics Education. 48(1-2), pp. 83 - 95. https://doi.org/10.1007/s11858-016-0758-z
Authors | Hoth, Jessica, Schwarz, Björn, Kaiser, Gabriele, Busse, Andreas, König, Johannes and Blömeke, Sigrid |
---|---|
Abstract | Rating scales are a popular item format used in many types of assessments. Yet, defining which rating is correct often represents a challenge. Using expert ratings as benchmarks is one approach to ensuring the quality of a rating instrument. In this paper, such expert ratings are analyzed in detail taking a video-based test instrument of teachers’ professional competencies from a follow-up study to TEDS-M (the so-called TEDS-FU study) as an example. The paper focuses on those items that did NOT reach sufficient consensus among the experts and analyzes in depth their features by coding the experts’ comments on those items and additionally considering their rating outcome. The results revealed that the item-wording and the composition of the group of experts strengthened or weakened agreement among the experts. |
Keywords | Content validity; Reliability; Expert-rating; Video-based assessment |
Year | 2016 |
Journal | ZDM Mathematics Education |
Journal citation | 48 (1-2), pp. 83 - 95 |
Publisher | Springer |
ISSN | 1863-9690 |
Digital Object Identifier (DOI) | https://doi.org/10.1007/s11858-016-0758-z |
Scopus EID | 2-s2.0-84962840605 |
Page range | 83 - 95 |
Research Group | Institute for Learning Sciences and Teacher Education (ILSTE) |
Publisher's version | File Access Level Controlled |
Place of publication | Germany |
https://acuresearchbank.acu.edu.au/item/8qw81/uncovering-predictors-of-disagreement-ensuring-the-quality-of-expert-ratings
Restricted files
Publisher's version
123
total views0
total downloads2
views this month0
downloads this month